deninet

I have been thinking a great deal lately about what do to about the Idea Database. After the long drive to complete the new Windowlight theme (now up and running on the site), this core feature of deninet 6.0 hasn't received nearly enough attention.

When I originally set out to recreate the system we had under deninet 5.0h and earlier, I'm confronted with two basic strategies. The first is to search throughout the Drupal module repository and attempt to find a solution that would create the necessary facilities to meet the original spec. The second is to write our own module from scratch. I've dabbled in module creation before, but ultimately found the projects I was going to undertake too complex or unnecessary.

Relying on contributed modules does have some serious advantages. Instead of a development process, the effort becomes largely that of configuration. Once the possible methods have been reviewed and compared, the list of possible methods can be reduced to an acceptable solution. After that, it's only a matter of implementation and adjustment to the new features and consequences. I've reviewed a number of modules that could possibly work, but the only one that met expectations was the Node Comments module. 

Node Comments is unique in Drupal as it replaces a bit of functionality provided by the core product. While Core Comments is packaged as a module, it is such an essential component that it is a dependency of several other modules. While this would provide a functionality similar to what I was hoping to achieve for the Idea Database, it affects a larger swath of the site functionality than I would prefer. Taking this option would have some serious drawbacks.

For that reason, I've been investigating creating a module to implement the functionality. How to go about doing that, however, becomes a discussion in of itself. 

The module could be developed in one of two ways. It can be created as an integrated, drop-in solution that would provide everything necessary in order to function. It would be an "Idea Database" module in the truest sense. This has advantages of integration as well as being a contribution to the open source community. It would be easier to implement add-on modules to enhance functionality later (Idea sponsorship and donation). The other method is to write modules only to implement features in a generic way. Instead of an iidb module providing Idea and Thought content types, there would be a "revision review" module that could attach to any CCK type. This would be more generic but would certainly implement the Idea/Thought topology originally drafted in the spec.

There is a line of thought that suggests if something is part of your core business, then you should develop it in-house. You shouldn't trust it to third parties that may not be in line with your vision. The Idea Database is part of what I consider as the core business of deninet. If it's really so important, shouldn't it be written in-house?

The problem, as ever, is time. Developing any sort of module takes a degree of time and effort that I could be putting toward other pursuits such as Paper Girl. With my time and energy limited as they are, I'm concerned that this "simple" module would require a lengthy development cycle.

At the moment, I haven't quite decided what to do. The clock is ticking for this years DOR submission, and I would not want to opt out if at all possible. It may be best for me to let the issue of the Idea database to sit for a while I work on my DOR submission. Perhaps the time away will give me some perspective.

The Windowlight theme made some progress this weekend despite my limited time. Since my last, hastily written entry, it dawned on me to check out drupal's themeable functions. While these functions solved some of the string-replace hacking, it's far from the proper separation of data and presentation I would prefer to see. 

This weekend I focused on two particular aspects of the new theme: breadcrumbs and error messages.

Breadcrumb trails are today considered a standard website UI element. They allow you to quickly navigate deep hierarchical structures by providing you a series of links that refer to "parent" pages. From my own experience, I know that no complex website should be without a breadcrumb trail element. In my previous theme efforts of Citylights and Starlights,  I had left the breadcrumb trails provided by drupal relatively unmodified. I was still new to theme development and as a result, did not put as much thought into their value. 

This time, however, I wanted to improve the deninet UI experience, and proper breadcrumb trails are essential. After some research at the venerable listamatic, I came up with a preliminary design. Implementing it in drupal, however, proved far more difficult than I had expected. The HTML provided by drupal wasn't ideal for this sort of modification. Unaware of the theme functions, I resorted to some ugly hacks to even try to get it to work. Even then, it wasn't as good as it could have been.

As I was clicking through Stumbleupon in a zombish attempt to stay awake, I ran across this article from Janko at Warp Speed. The article lays out how to create http://apple.com style breadcrumbs using some simple CSS. While my opinion of apple has suffered lately due to their poor attitude toward iPhone developers and 3rd-party systems, I still find their software UI standards to be the best around. Drawing upon the article and the theming functions, I set about coding a new breadcrumb widget.

During the process I began to find it difficult to find a spot to put the thing in the web design. I eventually narrowed it down to two locations. One was over the main header graphic, the other at the top of the content area. After some debating, I settled for immediately above the content area. It's closer to the text, meaning less mouse travel for users. Even with the issue decided, something felt wrong. It wasn't just mouse travel, but scrolling.

Previous versions of the theme had the header graphic taking up a majority of the screen space. This forces the user to scroll downwards just to see what a page is about. In other words, the graphic was unnecessarily dominating the design while contributing little to usability. For this reason, I shorted the header area by several hundred pixels. This reduced horizontal scrolling and tightened the overall feel of the theme.

Also, Windowlight will feature the return of something not  seen in the current Starlights theme -- the Denizen Entertainment logo. The "brick" didn't seem to have a good place in Starlights. I was also away from the notion as deninet as a organization toward that of a solitary effort. The logo didn't feel appropriate at the time, but I've grown to miss it more and more. Staring at the earlier versions of Starlight also felt...lacking. There was a distinct sterility I wanted to correct. Placing the logo in the corner of the content area was the obvious choice for placement, as it's be the standard location going back to deninet 2.0. Sizing, however, was a concern.

I didn't want to make the logo appear too small. A small logo almost seems apologetic. Too large and the logo would dominate the page. I wanted it to be just right, present and quietly proud. After some trial and error and encouragement from a friend, I selected the size you see in the image. Interestingly, it also solved some positioning problems with the breadcrumb trails. 

As I was working with the new theme -- testing it out on different pages -- I also began to find another problem. The breadcrumb trails provided by drupal aren't particularly effective. More times than not, I would find the trail to be "home > Current page", a nearly useless configuration. After some searching, I located the Custom Breadcrumb module, which allows you to control your breadcrumbs. Combined with views, I believe the usability of the site can be greatly improved.

Today, however, I wanted to cover a different series of issues. For a while now, I've wanted to customize the status and error message boxes provided by drupal. In previous themes, I've simply styled them without seriously examining their usability. In deninet 4.0, I made a special effort to create friendly error pages. (This incidentally, also spawned one of my favorite error messages ever: "Uncaught FeatureNotImplementedException, explaination: programmer lazy.") Drupal provides two kinds of error messages, status messages and error messages. The only difference between the two in Drupal's default presentation is that of color. Status messages are green, error messages are red.

I felt this could be improved through the use of the theme functions. I created a override that interpreted the error type as it is sent to the theme engine. Then, it would either print "The following action has been performed", or "The following errors occurred" in addition to a list of what actually happened. It was easy to add an additional line at the bottom to suggest to the user how to react to the message.

Oddly enough, I spent most of the day trying to get the background of the message boxes just right. Originally, I wanted to include a simple icon. As I began working on it, however, I felt I wanted to give it more than a simple solid-color background. Windowlight has a recurring theme of stripes throughout its UI and I felt that the same could be used here for a subtle, but aesthetically enhancing effect. The stripes were easy enough to produce, but the message boxes still lacked an icon.

There was no easy way to add one without adding enough div to the message. While I could easily do that, I stumbled upon another problem. I had wanted to identify status messages with a round 'i' icon, and errors with a triangular or octagonal '!' icon. As you can see, 'i' and '!' next to each other are kind of difficult to tell apart. After some more thinking, I decided to go with large, subtle text in the upper right corner of the boxes.

The theme is shaping up. Whenever I go back to the production site after working with the new theme all day, I shudder a bit. Hopefully, the new theme will be ready soon so I can focus on competing the 6.0 features.

As a departure from previous site upgrades, I'm stressing a high amount of usability in deninet 6.0. By nature, I tend to be a UI snob, preferring to tweak and tune aspects of the user experience that would make everything easier and more intuitive. While this is commendable, Drupal itself is not making this easy.

Creating a strong UI experience was a key aspect of deninet 4.0. I was able to tweak nearly everything in the experience. Form organization, buttons, site layout, and so on. At the bottom of nearly each page was a two column section labeled "Where to go", providing links or actions for the current page. Each option was explained with clearly written text. While I made the blunder of using javascript links as opposed to standard Submit and Reset form buttons, I loved the site I had created. Deninet 4.0 was a clear evolution from versions 2 and 3, and easily the most refined deninet experience yet created.

While working on deninet 6.0, I pulled up my archives and reviewed my ideas from that period. I began to think, Wouldn't it be nice if we could provide the a strong UI experience for the next version? The biggest problem, unfortunately, was that tweaking that experience is no longer a simple matter. All versions of deninet prior to 5.0r were build on a custom content manager (or no content manager at all). I had maximum control of each individual page. If I wanted to implement a particular feature or change the presentation of a page, it would simply be a matter of making the necessary code changes. 

Now that we are using Drupal, things aren't nearly as easy. For example, I recently had an idea to modify the Create Content page to be a table of icons and friendlier descriptions as opposed to the current definition list of links. To my annoyance, there appears to be no way to nicely modify this page short of ugly theme hacks or module development. Web searches show that few people have even considered such a thing. I'm not even sure it is possible to write a module to modify the page. It's possible to supplant the entire page with a custom one, but without hooks into the content types, it would be no better than a static page. 

Block and breadcrumb themeing is particular annoying, as both are provided to the theme as a scalar variable containing HTML. If I'm unhappy with the output, I have to resort to string or regular expression replacements that would execute for each page display. If that doesn't work, the only means available to me is complex string manipulations and HTML parsing. As someone experienced with PHP development, I'm sensing a serious lack of separation between the data and presentation layers.

It's possible I will have to accept what I can when it comes to UI in deninet 6.0. As long as I continue to use Drupal to power the site, I will need to continue researching just how far I can modify it. 

I have been attempting to do more than meander when it comes to the deninet.com upgrade. Unfortunately, I'm running into a lot more issues than I expected, even when attempting to upgrade from Drupal 5.5 to 5.10. 

Throughout the upgrade process, I am looking for "blocking issues" -- situations or errors that currently have no immediate or easy solution. One of the biggest I've run across is the lack of support for the CRAM module, which helps to secure our logins. The modules allows an encrypted hashcode to be passed in liu of a plaintext password. Granted, it's no match for a tried-and-true security solution such as SSL, but I currently do not have the $80 required to configure it on our hosting provider.

Right now there's a debate between myself and the other active staff members on this issue. Overall, we want to set up SSL, but the money simply isn't there this month. Without CRAM, our passwords will be sent clear over the wire. A temporary workaround would be for all deninet users to change their login with an "untrusted" password. This assumes that you are using the password only for the site, so that it cannot be captured and used for other internet accounts. I'm not terribly fond of this method, and I'm still investigating other options.

Windowlight DR5In the meantime, I've put some work into a new theme for the site using a different method. The previous two themes for deninet, Citylights and Starlight, both use a CSS technique called "fluid layout". Fluid layouts have the benefit of using all the space provided by the web browser. Creating a visually rich liquid layout, however, is fraught with difficulties and pitfalls. The previous two themes have significant problems and hacks in order to make them work on Firefox and Internet Explorer. I've often had to make unfortunate trade offs to make them work.

This time, I wanted to avoid the vast majority of those problems while making the theme visually rich. After some research and copious examples from various websites and blogs, I decided to try a "fixed-width layout". Unlike a fluid layout, a fixed layout does not expand or contract with the size of the browser window. Instead, it stays the same width all the time. Typically, these themes are centered, so that the content realigns itself to middle of the browser window horizontally. If the website width is less than the browser window width, the user will be forced to scroll. 

While developing the new theme, "Windowlight" (see preview on the right), I quickly discovered that creating a fixed-width layout corresponds better to my preferred web design methods. It reduces the number of hacks, image slices, and other things that are endemic to fluid layouts. The design degrades nicely to a limited set of large images, making it easier to use photo-manipulation techniques to achieve new graphic effects. This does not mean that fluid layouts are horrible things that should be avoided, it means that they work better for me, and better for deninet. 

Currently, Windowlight draws from my 2007 submission to the Transgender Day of Remembrance Webcomics Project. I still like the piece and consider it the best I've yet produced. However, I'm concerned that my repeated use of this image (it's my desktop wallpaper) I may quickly tire of the new theme. Furthermore, I am not happy to see Akisa depart her presence as part of the web design. Given my complete lack of drawing output this year, I very much doubt I'll come up with a new graphic any time soon. 

I've tinkered with other pieces of artwork in my library for use in the layout. One such attempt was with "Answer" -- a not terribly subtle bit of Akisa art. I very much doubt I will use it, as I do not consider deninet a horror media site, and am dubious about the shock value. Other graphic I've considered is the incomplete "Akisa Listening" graphic I had drawn years ago. I like the graphic as well as continuing the motif of Akisa and her portable music player. The problem is that this image is hand drawn, and will have a grainier appearance when scanned and colored compared to the DOR image.

In addition to working on the theme, I've spend a great deal of time investigating what new features to add to the site. I have already created a new event system, allowing members to create their own calendar and list of appointments and meetings. These events will be tied into each individual channel, and viewed collectively on the Springboard. The Springboard is a user dashboard allowing users to view all their events, submitted Ideas, as well as keep up to date on their subscribed channels. This feature is currently available to registered users, but is not yet complete. 

The majority of my development time has been geared toward recreating the Idea Database. I've investigated several different options for creating the Idea, Version, Thought, Vote hierarchy that was available in deninet 4.0 - 5.0h. It's been long decided to use Drupal's built in revisioning system to create new Idea versions. There are several useful modules that attach to this mechanism, including one that shows the difference between selected versions.

According to the original Idea Database design, Thoughts are attached to each version, not to the Idea itself. This way, each new version has a clean slate for development. Currently there is no way to implement this feature short of writing code. I've wanted to avoid this because I have only limited time to support a programming project. Right now, the solution seems to be heading towards the Node Comments module. This module allows comment nodes to be replaced by any CCK type defined by the site administrator. Comments can also be displayed through Views for dynamic ordering.

Node comments, however, are not without their down side. The module is incompatible with three key modules within the Drupal core -- Comment, Forum, and Tracker. We will lose all of the some 149 comments currently applied to content nodes. I may be able to migrate them to the new Node comment system, but this has yet to be tested. While we have forums on the site, there's not much to keep and the functionality can be (possibly) created using CCK and views. Tracker provides a list of your activity on the site on your user profile. This functionality can easily be recreated using views and integrated into the Springboard -- a more appropriate place for it anyways.

In order to discuss these options with other deninet staff members, I've created a new beta site. The site is currently only available to current deninet users. The beta site serves both as an online backup and as a proving ground for new deninet features. Members can also give the new theme a spin, keeping in mind that it is not yet complete.  

A developer's prerelease of the "Windowlight" theme intended for deninet 6.

I have been slowly updating deninet over the course of the last few weeks. I was hoping to be at v5.3 by now, but the process has been meandering in practice. Originally, I had intended to practice a complete backup on the test server prior to running a complete upgrade on the production server. I've gone through this process before -- from 5.0h to 5.0r, and again from 5.0r to 5.1. Not only can it be disruptive, but tedious.

Adding new features has been foremost on my mind. Making upgrades doesn't rank all that high on my scale of enjoyable tasks. I have a day job to support and a rigorous workout schedule to support leaving me little time for deninet or fun. Deninet can actually be a great deal of fun for me, but I have to pick my battles in the 2 or 3 hours I have free each night. Developing new features is simply more fun.

This is not to say that some upgrade work hasn't been done. I have updated the modules that power the website backend. The content manager core has yet to be upgraded, a task I should complete sooner rather than later. The module upgrade process is actually quite time consuming itself.

Each upgrade requires a review of all modules powering the site. If some didn't work out, or were underused, they may be removed in the next release of deninet. Furthermore, alternatives need to be researched and tested for each module. Newer modules may implement features in a more efficient way, or provide a better user experience.  Once reviews are complete, updates for each module are downloaded, installed, migrated, and finally tested. This takes a lot of time. Furthermore, as the sole developer for the site, I'm the only one available to test. This can result in missed details or use cases that I wouldn't have imagined.

I've been thinking increasingly often of what's next for the site. While there's nothing technically wrong with deninet's current incarnation, the underlying purpose is in serious need of refinement.

Deninet 1.0 through 3.0 were geared  toward content delivary of first Denizen Soft's, and then Denizen Entertainment's projects. Due to various circumstances -- life, work, my own ineptitude -- none of those projects have yet come to fruition. Toward the end of deninet 3.0, it became obvious that we needed a proper framework to help us create, manage, and publish our projects. 

Deninet 4.0 was the birth of "The Internet Idea Database", a web application geared toward communal development and tracking of ideas. I spent a great deal of time on Gazelle, the software powering the Idea Database. I still love the concept of communal idea development including our sales tagline of "Invent, Share, Expand".

Gazelle wasn't the most complicated software project I've attempted (InterLock still holds that title), although it had plenty of problems of its own. In Gazelle, Ideas (big 'I') were a fundamental unit of content. Ideas were made of Versions, and contained the actual content of the idea itself. Versions were submitted to the community; authors and others contibuted Thoughts to extend or refine the Idea. Those who liked a particular thought could vote on the thought. When the author is satisfied, they considered the submitted Thoughts and published a new Version. Authors weren't required to change their Idea based on community suggestions, they retained complete control of the Idea's development. Ideas could be grouped at a higher level via Channels, which often reflected a sense of a whole project. 

On paper, the concept looks wonderful. Individuals and organizations can post and track ideas throughout the entire development lifecycle. Once we had a working version of Gazelle, the Idea Database became a useful tool for my own projects. There were, unfortunately, a huge number of problems with the system. There was very little security within the framework. All Ideas were left wide open to the entire site. This reflected my high-minded idealism at the time,  today I just shake my head my nievate. Gazelle itself was a nightmare to extend. Each new content type required new code. Everything was library and no engine -- there was no dynamic way to organize content display. 

The biggest mistake I had made with it, unfortunately, was I began to think of the Idea Database as a generalized content manager. News and blog posts co-mingled with Ideas proper resulting in confusion. In retrospect, I probably did it to reduce the amount of content types within the framework as well as gain the functionality Ideas provided.

Six months after the "completion" of Gazelle, I dumped the entire project. I hated to do so after investing nearly two years of my evenings developing the thing, but I felt it was really the best choice. I had come to the realization that I'd never be able to keep up Gazelle while attempting to work on Paper Girl. There were fundamental problems with the design, and I didn't have the knowledge, expertise, let alone time to write them myself. After investigating my options -- as well as rewriting the framework entirely -- I selected the open-source content manager Drupal to power the next version of deninet.

Deninet 5.0r was the first version of the site to run the new content manager. The site also completely lost it's Creative Cooperative bent and became my own personal domain. My failure to perfect the Idea Database and attract interested people to the concept led me to this decision. Interestingly, once the whole "Were a real company, honest!" charade was dropped, the site began to swell with content. Instead of being project or idea centric, I uploaded my artwork, some writing, and began writing posting blog entries to the main page. I could, of course, publish more complex creative works from the site if and when I ever came to that juncture.

Then a curious thing happened a few months ago. I was no longer the only active person on the site. This both caused me excitement and concern. First of all, deninet 5.1 was designed to be a single user site. While I kept multiple users in mind, there were critical places where it began to break down. The image gallery, for example, isn't set up for multiple users. Even though blog posts are nicely separated, the "river of news" main page is not terribly effective. 

Even more curious was the fact that I was no longer the only person wishing for a Creative Cooperative. Indeed, there seems to be a strong desire to refine the site and reclaim the purpose I had thrown away with deninet 4.0. Although the desire was there, I haven't really been in the position to research our options.

Looking back at the history of the site, certainly have gained a sense of what was good, and what was a mistake. Right now, I'm looking at the Idea Database topology I described above with scheming eye. Yes, it was a mistake to shoehorn all content management into the concept. It may have also been a mistake for me to write the management system myself. As one person, I can only write so much. Developing a new CMS from the ground up simply takes too much time. Gazelle's internal lack of generalization also complicated things unnecessarily. Switching to Drupal, however, saved me huge amounts of development time and added many new features.

What if, I began asking myself this afternoon, I used Drupal to power the Idea Database? We would certainly gain a powerful backend with a preexisting development community. We would only be responsible for what fulfills our goals as an organization. Are Ideas as laid out in the topology, I thought, enough to fulfill the ends of a "creative cooperative"? This is a trickier question. While I believe that the Idea Database topology will be part of our end goals, I very much down it will be enough to fill the project management role required to bring make ideas manifest. There's a lot I have yet to investigate.

I very much doubt that deninet will jump from 5.2 to 6.0 directly. There's a lot of new technology and refinements required in order to get to that point. The present site needs to be updated to the latest version of Drupal and key features made available to all users. There are several more minor releases in our future before we get that far. I for one am very much excited to see how the site will evolve.

Deninet is currently experiencing periodic outages due to possible hardware problems. I'm beginning to think it's a failing drive, but I won't be able to know for sure until I'm back in Minnesota on Friday. 

I discovered recently that Drupal 5 does not encrypt the password on login. I cconsider this a serious hole in security as a third party could intercept communication and capture the password.

Ideally, we'd enable SSL access to the site so the line is encrypted. This cannot be done right now due to the aging OS on the deninet server. Setting up SSL on Gentoo is a mess. For now, I've installed the CRAM module to at least patch the hole. This module sends a random token to the user. The token is used in combination with the password to generate a MD5 hash using Javascript on the client side. This hash is what's sent over the line -- no plaintext passwords. The server then takes the token and looks up your password and generates another hash. If the two match, you're logged in.

This isn't the greatest security, but it's better than nothing (which is what we had before). MD5 hashes can be broken fairly easily, and the token can be intercepted on send. I hope to keep this module around even after implementing SSL, thereby making it much, much harder to break. 

I'd like to deal with the issue of the aging OS and lack of SSL in one fell swoop. In order to do this, I plan to wipe the deninet server clean and reinstall Ubuntu Server 7.10 on it. User home directories, including the one on which the main site is stored, will be preserved as they are on a different disk. Since I will be out for three weeks starting Sunday, I'd like to take care of this ASAP. 

Please be advised this will create some downtime while I reconfigure the system. 

The "brick" in glorious print resolution, and SVG.