Enterprise Drupal @ SF State
Choosing Drupal Began with search for WCM for DoIT Tested many (Plone, OpenCMS, Joomla, etc.) Settled on Drupal for functionality and a11y Started w/ 1 virtual server, no real environment Started with 5.x and moved to 6.x Campus already had some Drupal exposure
Ancient History Portals required on CSU campuses •
•
Project Approach (IBM Websphere) –
Collaboration between SFSU Team and IBM Team
–
Approach according to the established IBM Method
An emerging consensus regarding portal development includes the following major best practices and considerations: –
There should be one AND ONLY ONE horizontal portal on campus;
What happened previously •
IBM Portal took years to implement base features. Required constant maintenance and time-consuming PMRs.
•
Other groups/depts. Didn't fully buy-in, saw IBM WebSphere as a “black box”
•
Project Manager/Maintainer left
•
Said hey “How about going to Drupal, they have it running successfully for WCM, would maybe take a couple months to switch”
IBM Lotus WCM vs. Drupal •
Java
•
PHP
•
JDK/JavaEE/JSR 168/WSRP
•
drupal_hook API
•
Services module
•
Portlet Factory
•
Any IDE/Editor
•
Monolithic
•
•
Broke weekly
Core is relatively small
•
Few expertise
•
Huge community
•
No community
•
•
Only support is PMRs
Support via forums, IRC, Acquia
The A Team •
Me – Architect / Developer / Themer
•
Supakit “superkid” Kiatrungrit – Developer
•
Bora Kou – Systems Admin (Part time on this project)
•
Jason D'Silva – Student Asst. Developer (10 hours/week)
•
Management position was actually vacant during our 2.5 month sprint. So had a parttime interim project manager.
Scrum style agile development •
What we liked: –
Quick turnaround
•
What we didn't like: –
Scope creep
–
Weekly feedback
–
Loss of resources
–
Clear deadlines
–
–
Daily checkins
Cramming at end
–
Learning a lot
–
Not enough time for testing/QA
–
Working a lot
Demo
[demo] of SF State Drupal Portal http://inside.sfsu.edu/portal http://drupal.sfsu.edu/drupal
Architecture •
2 Dedicated RHEL 5 boxes run Apache (Xeon 2 ghz + 4 GB RAM)
•
2 Dedicated RHEL 5 boxes run MySQL (Xeon 2 ghz + 4 GB RAM)
•
1 Dedicated box runs MySQL cluster mgmt node (+ MMM replication)
•
Load balancer in front + 2 mysql db connection scripts Big IP Load Balancer
MySQL DB connection test script
Apache+PHP 5.2
Apache+PHP 5.2
MySQL Cluster 7.x
MySQL DB connection test script
MySQL Cluster 7.x MySQL Mgmt Node
Drupal & MySQL Cluster :( Drupal will not run out of the box on ndbcluster tables Problem: menu_router table and locale_sources tables A bug was filed with MySQL, rejected because ndbcluster doesn’t support row lengths of that size Issues on d.o. filed to address this in core http://drupal.org/node/391130 and http://drupal.org/node/703916 Split the menu_router table into smaller, more manageable tables Remove the key reference on locale_sources or change the key data type to be something other than text/blob Workaround at http://www.coderintherye.com/
SF State Custom Modules User Interface Modules
Utility Modules
Address change
connectioncheck
Benefits pps
deletecookies
Distribution Groups management
encryption
Emergency contacts pps
langhide
Ethnicity resurvey
Scheduled message
Financial Aid Offer Letter
Show more filter
Financialaidofferletter admin
Shibboleth data mapper
Military status
sfsulogout
Name Change Self Service
shibbolethdata
Nextmuni
validator
Official Transcript Request
xmlservice
Getting the Campus Community to Develop
Offered two trainings –
1) How to install Drupal on your localhost
–
2) How to create Drupal modules
–
3) (Future) How to create web services
Need hand-holding Monthly meetings and code workshops
How to get a Module (portlet) into Portal First content should be discussed with the oversight group before beginning development to ensure Portal is the correct place for it Some departments or groups may have a page in Portal that they are responsible for and control the content for All changes must go through the development cycle including unit and load testing Security testing of code (Plug for “Cracking Drupal�) Clean up and review of code
Development cycle Develop code Run unit tests • Check into SVN • •
Additional development Interface testing • Tag in SVN • •
Verify automated tests User acceptance testing • Content is added directly here • Management approval • •
Automated tests run If any tests fail, there is no push to production • •
Automated tests If any tests fail, system is rolled back to previous build • •
Prepare Define requirements including functionality and UI Remember you are working within the Portal template so many UI elements are predefined
Define interfaces to backend systems The most common interfaces are ď Ź
ď Ź
REST/ WebServices that should follow w3 standards LDAP (V3)
Determine where portlet will be located and who should have access
Local development Build portlet Run unit tests Unit tests should include all UI components and all backend interfaces
Check into svn Request dev build
Backend Work with the data and application owners of the source systems to build the required backend interfaces Make sure you follow the defined security model if there is one, or work with us to define one if the system you are connecting to has no model defined
Set up web services
Development (DEV) Run unit tests Run integration tests, finalize development Peer review all code Tag SVN code for test Request Test build Code is pulled from SVN into Dev
Test/ Quality Assurance (TEST) Define test plan and execute all functional tests Tests must include all functions, UI components, and backend interfaces Perform user acceptance testing Perform load/performance testing Submit automated test suite for Staging and Prod This test suite must be comprehensive Make any content only changes directly in Test A manager must sign off indicating the following: He/she is aware of all changes going into production and approves them for the specified date The automated tests sufficiently cover all functionality All code has been peer reviewed The load/performance tests prove that the application meets the minimum requirements Tag code in SVN for roll out Code is automatically pulled from TEST at 1am and deployed into staging
Staging Automated test suite is run If any tests fail, production roll out is not performed If all tests pass, promotion to Prod occurs automatically at 6am
Production (PROD) Automated tests are run If any tests fail, production is rolled back to previous release
Accessibility
508 compliant
All pages were run through manual checks
Theme is color contrast compliant
Testing out using the accessible_content module developed at CSUMB
Security
IP based Access for back-end services
Everything is behind a firewall
Everything is behind Shibboleth auth
Core PHP Module is removed
Role based access to pages, panels
Daily log checks (in future, archive logs in central change mgmt repository)
Grendel-Scan
OWASP standards
Scalability •
We are not very big Handled ~500 concurrent users without problem
•
Peak ~5,600 visitors/day
•
357,000 visits and 1.7 million page views from March-May Shibboleth single-sign on & Google Analytics
•
Servers are way underutilized, use about 15% cpu on norm
Performance •
Initial Build took 7.5+ sec. Load times
•
Used Xdebug to find drupal_http_request was problem. Used cron to cache
•
Down to 3.5 sec load times.
•
Installed APC via PECL (quite easy)
•
(Make sure to give it enough memory)
•
Down to ~1.5-2 sec load times.
•
Ensure Oracle db connects quit if can't reach (need to upgrade to 11g)
Where we are Now vs. Where we want to be
Now campus websites are fragmented, outdated Not all sites use Univ. Template, and some don't use it correctly or update it timely
Depts want to move to central hosting We have a monthly meeting Drupal User Group
Depts. want to put sites in centrally managed highly available Drupal install for easily keeping sites up to date 1 Drupal theme means a consistently updated look Need installation profile for point & click
Tools to consider •
Aegir (still maturing when we looked at)
•
Open Atrium
•
Pressflow
•
Memcached/Boost
•
Cloud hosting via Acquia/AWS or Rackspace as well as consultants
Drupal is not the solution for everything (though I love it)
Drupal is not –
our document management system (though it could be someday) We use Sharepoint
–
our project management system (again Sharepoint)
–
our help desk ticketing sys (remedy, gia)
It can be made to do all these things, but other products are out there
The Importance of Community
Drupal.org
Bay Area / Berkeley / South Bay DUGs
IRC: #drupal-support or #drupal
On campus: http://drupal.sfsu.edu/drupal
Monthly Drupal meetups (now workshops)
California Higher Education Group: http://groups.drupal.org/california-highered
California Higher Ed Drupal Sites •
California Institute of the Arts
•
CSU Monterey Bay portal (login required)
•
CSU Monterey Bay Catalog
•
Moss Landing Marine Laboratories
•
San Francisco International Airport (SFO)
•
SF State University Portal (login required)
•
SF State Drupal Sites (login required)
•
Stanford University Sites using Drupal
•
UC Berkeley Drupal Sites
•
UCLA Graduate Students Association
•
UCLA Semel Institute
•
UCLA Family Commons
•
UCLA School of
Questions?