New Web Languages and Frameworks (and a Brief History of Web Programming): Pt. 1

Software development has changed so much since I first got into the business over a decade ago, and lately, sometimes I feel like I’m becoming that 50 year old dinosaur who sits in the corner mumbling to himself about “the old days” of programming and how all the new technologies are ruining “proper'” software engineering. Ok not literally because unlike those guys (and we’ve ALL worked with folks like that) I love adapting and learning new technologies, but metaphorically…yup. I have my comfort zone which consists of Java and .Net, mainly on the Windows platform, with Windows centric middle-tier/backend plumbing:

If I had to classify myself and what I like to develop, it would definitely be backend/middle-tier stuff. I am by no means the best front-end designer (WPF or Web stuff), but can get it built if a designer hands me the CSS and HTML framework/storyboards. A little background (queue harp music here);

When I first got into the business, you were either a Java person, or a Microsoft person.Third party languages/frameworks were non-existent on the LAN (outside of Perl, which was the precursor to modern day frameworks via CGI (Common Gateway Interface)…if you’ve ever seen /cgi-bin/ in your URL, it was probably a compiled Perl script doing the dynamic processing. CGI is long dead, thank God. I wish Perl would suffer the same fate.

I did a little of both initially. If you were a Java person, there were few frameworks outside of the Java libraries themselves, and if you were a Microsoft person, you used either Classic VB (v. 6 in my case), C++, or ASP if you dared to venture into this brand new thing called the web. At the time I thought I was in heaven: I loved C++ but it was way too complicated, so I went the VB route (which made ASP that much easier to pick up since VBScript is a subset of VB). Anybody remember Visual Studio 6? Visual InterDev (what a piece of crap IDE)? SQL Server 6.5? If you never worked with any of those, count yourself lucky.

Then the web (HTTP/HTML) started to take off on the LAN. Java came out with some frameworks, mainly JSP. And, in 2001 Microsoft rolled the dice with this entirely new framework called .Net, with the promise that given a unified framework, anyone who could pick up the new languages (VB.Net, which had absolutely nothing to do with VB6…there were a TON of pissed off VB6 developers as VB.Net was 100% backwards incompatible with VB6, and holy crap, now VB developers had to learn this thing called Object Oriented Analysis and Design) could write apps that targeted the web AND the desktop, without having to write much Javascript or HTML by hand, the Asp.Net libraries would generate it all for you vie WebForms. For me it was a godsend. I immediately jumped off the VB6/ASP bandwagon, ditched VB entirely and learned C#.

Initially this was great…just use the built in frameworks, hammer out some code, et voila…a website (back then, corporate LAN websites didn’t need to be pretty, just functional). For the most part (outside of god awful non-standards compliant HTML) it just worked. The LAN didn’t care about standards (though they most certainly do now), or dynamic GUI’s, or postbacks/page reloads on every request. Everyone was happy. The deployment teams had a centralized point to roll out updates (the web server), the support teams were happy since they didn’t have to troubleshoot hundreds of machines running custom windows apps, just the web server, and the end users were happy because the web is so much more intuitive and consistent than having to learn some home rolled windows app, which probably had zero consistency with other home grown windows apps.

This worked great until about 2006…then the web blew the absolute hell up, and lots of independent developers started realizing how they could tame the web and make it do some pretty incredible stuff that no one expected possible. Browsers themselves became programming platforms in their own right via CSS and JavaScript (and other cool client side technologies). This was a double edged sword though: The benefit was that the web became a truly viable programming platform, with all the richness of client applications, without all the headaches involved that I mentioned above. The web became less about HTML and displaying information, and more about passing information around and letting the browser figure out what to do with it. The shift over the past 20-30 years has been pretty wild: We have gone from thin client (Mainframes and Greenscreens) where all the processing is done on a central server, and dumb terminals have to play round robin for processing time. Then thick clients came along: Client based applications where all the processing is done on the client itself (PC’s) with minimal processing on the server (probably just a database repository). Then web 1.0 came along, back to thin client architecture: The centralized web server does all the processing and spits out some HTML, all the client has to do is fire up a browser and render the HTML to the user. Now we’re going back to thick client (though it’s more of a hybrid) where the web server process chunks of data, does some processing, send out these chunks of data fragments to the client, and the browser can handle a ton of processing via the new JavaScript frameworks that exist (more on that in a bit). I expect this latest trend to continue and advance for some time. The downside? It is extremely complicated with all the layers involved, and web developers are swamped with all the disparate technologies, and how to glue them all together. Most people don’t realize just how complicated websites are these days. And just how much work goes into creating even a simple web 2.0 site. A typical web developer needs to at least be functional (if not an expert) in the following technologies (and this is a cursory list, no doubt there are plenty more that can be added to it). I’ll start from the backend and move up:

  • Database design: Some frameworks will take care of this for you via ORM’s, but this only works for small/medium size sites. Some developers used to devote their entire careers to just this alone…this is no longer the case. This includes:
    • Some dialect of SQL (Structured Query Language), a programming language based on Lambda Calculus. Just to make things interesting, each database vender has its own dialect of SQL, so it usually is not portable (even though an ANSI version does exist, vendors add their own extensions)
    • Database administration. This means learning hardware, RAID setup, roles and security, backups/restores, replication schemas, performance tuning, physical/logical partitioning of database files, production troubleshooting.
  • LAN technologies/operating systems/network theory. This is where things can get really hairy as most organizations have a dedicated infrastructure team, however developers are expected to have a pretty thorough understanding of the following infrastructure stuff:
    • Directory Services of some sort (AD/LDAP/RADIUS/etc.)
    • Domain Name System (DNS) and Subnetting
    • Dynamic Host Configuration Protocol (DHCP)
    • Basic network protocols: Transmission Control Protocol (TCP), Internet Protocol IP), and HTTP (which sits on top of TCP/IP…this protocol is what made web developers lives so miserable in the “old” days because it’s stateless, unlike TCP which is stateful), File Transfer Protocol (FTP), Simple Mail Transfer Protocol (SMTP)
    • Basic knowledge of networking hardware like routers, switches, VLAN’s, hubs, NICs
    • Client operating system configuration for the above protocols
  • Middle tier layer frameworks/technologies: This list alone could consume an entire post. The basics are:
    • Caching hardware: To offload client load on servers
    • Firewalls: These can wreak havoc on distributed systems if not configured properly
    • Web farm hardware/software: This is used to make multiple physical web servers act as one logical server, to offload server load amongst numerous machines. Big sites like Google, Amazon, eBay have thousands and thousands of servers around the world working together to server up web pages and crunch data as quickly and efficiently as possible. Getting these servers to talk to each other and act as one is probably the 8th wonder of the world.
    • Queuing hardware/software: The last thing a website wants is a dropped request/response, so this layer is in charge of “queuing” data if the servers are overloaded, and in the case of distributed systems, they guarantee delivery of a chunk of data, regardless of latency.
    • Web servers themselves: The big players are Apache and IIS, though there are hundreds to choose from. This is one of the few administration areas that developers actually own. An understanding of security, the HTTP protocol, caching, partitioning, configuration are a must. This is one area that drives developers absolutely crazy, but it’s a necessary evil as this is the heart and soul of a web developer’s application. Without a properly running web server, our apps are dead.
  • Though not necessarily a layer, the language and platform itself that will be used to write the application. Again this list could go on for pages and pages, but these are the major players (and the jist of the post, which I’ll go into at the end):
    • .Net: Two major web framework ship with .Net itself, and there are many others to choose from. Asp.Net WebForms, and Asp.Net MVC (Model View Controller). WebForms owns the LAN, MVC owns small to medium outward facing sites for the most part, but is catching on on the LAN (I prefer MVC).
    • Java: I’m a little dated, but JSP (Java Server Pages), Struts, Spring, and JSF (Java Server Faces) are the most popular.
    • Ruby on Rails: Ruby is the programming language, Rails is the framework. Ruby has been around for quite a while, Rails is fairly new to the scene though. It is a RAD (rapid application development) framework which mandates using the MVC pattern, and it took the web by storm when it was released. I’ve used it a bit in the past, and it is extremely powerful, very easy to learn, and it doesn’t take much work to get a fully functional site up and running, outside of learning ruby, and the rails framework. A quick list of sites that use Ruby on Rails:
      • Twitter. Yup. Twitter.
      • Yellow Pages
      • Hulu
      • github
      • Soundcloud
      • Groupon
      • and thousands more
    • Python/Django: A direct competitor to RoR, Python is the programming language, Django is the framework. Django is similar to Rails in that it mandates MVC (seeing a pattern here?), and again Python has been around for a while, Django is a fairly new framework. It also has taken the web by storm mainly due to its simplicity, and Python is drop dead simple to learn (if you are versed in any OO language, you can pick up the syntax in a couple of hours). Notable sites running Django:
      • Disqus (the service/site that powers my own site’s commenting system)
      • National Geographic
      • Washington Post
      • The Onion
      • Pinterest
      • Layar
      • NPR
      • New York Times
      • (lots of news sites as you can tell) and many more
    • PHP (which is a platform itself in its own right): This is the only popular web language that I know virtually nothing about, other than I really like its syntax as it's very C based. From what I’ve read, most of the popular frameworks for PHP pale in comparison to Rails and Django. What PHP really has going for it is turnkey CMS solutions, which is my next bullet.
    • Coldfusion (CFML): On its way out, if its not already dead. My first gig was in a CFML shop, and back in the day you couldn’t beat it. But, Adobe couldn’t keep up, and I haven’t seen a line of code in CFML in well over half a decade. Definitely on life support.
  • Just to further complicate web developer’s lives is that there is a whole slew of turnkey CMS/Blogging/Website solutions…many of them open source and free (you just have to learn the system, and pay for hosting), and a few proprietary and horrifically expensive. This is a layer that sits on top of the framework/language, and are useful if you don’t want to write your own solution, and is the last step before the content reaches the web browser. A quick list, the framework it’s built on (if at all) and the programming language:
    • Sharepoint, Microsoft Asp.Net, C#. This is for the LAN ONLY as it is tightly coupled to the Microsoft platform. A rudimentary SharePoint implementation will set you back 6 figures in licenses, which is why it is usually only used by large companies. It pays for itself though, and is a product I specialize in. (note: Microsoft does offer a stripped down free version called Foundation, which includes a free version of SQL Server called Express).
    • DotNetNuke: Microsoft Asp.Net, C# (was VB.Net for a long time). DNN itself is free, but to get any sort of functionality out of it, you have to buy add-ons and plugins, which can be very expensive. It is open source (but customize to your own heart’s content). Extremely customizable, fairly easy to use, and an extremely active developer community.
    • WordPress: PHP/LAMP (Linux Apache MySQL/PHP) Stack. Arguably the most popular blogging package on the web.
    • Drupal: PHP/LAMP. Probably my favorite CMS system due to the power of it. runs it, and it is probably the most widely used CMS (Content Management System) software used on the web.
    • Joomla!: PHP/Lamp. Starting to surpass Drupal as the most popular open source CMS solution.
    • Movable Type: Perl. Probably the only thing Perl is still good for.
    • Plone: Django/Python. List of sites here. Very powerful CMS framework.
    • Surprisingly, not many turnkey CMS’s targeting Rails/Ruby worth mentioning. The above systems account for > 90% of the popular vote.

Quick interjection: My site runs on a CMS called Community Server (which is now defunct and no longer actively developed by Telligent Systems, who morphed it into a proprietary product called Telligent Community which is no longer open source). My site is showing its age, but it still works, and I've highly customized many of the subsystems. Moving to a new platform just doesn't make any sense.

  • If as a web developer you choose to go with a turnkey solution, job pretty much done (given you have a good understanding of the previous bullets) outside of finding the plug-ins/add-ons you need, doing some configuration in your CMS’s control panel, probably implement some custom JavaScript/CSS, design some fancy graphics, then deploy. If not, it’s time for the last step: Client side browser development technology choices and implementation. Again this is another bullet that could take up an entire post, and in this case, that’s exactly what I’m going to do in Part 2 of this article, which I’ll post later.

These bullets encompass about half of what it takes to be a web developer. The other half (and by far the most important from the end user’s point of view since all they care about is how it looks, how easy it is to use, how responsive it is, and the overall “hook” and feel as I like to call it: You gotta get them to come back, and tell their friends about it. The end user doesn’t care about the above, but it’s just the tip of the iceberg. What most users don’t know is just how much work goes into creating a fully functional website. I’m not out to get them to care, but to simply A) let them know that it’s not just about typing up HTML pages anymore and B) hopefully get some sympathy from my fellow web developers as to just how swamped our brains are with ALL THIS STUFF!

So in part 2, I’ll touch on all the various client side technologies that web developers are required to know (in addition to the above, and the underlying server side language which is what most programmers spend their lives trying to master just by itself), why these frameworks exist, the rise of web 2.0 (even though I loathe this term) and I’ll touch on NIH (Not Invented Here) syndrome, and what these new frameworks have done to change the face of web development: Even though they’ve eased the lives of web developers, they’ve forced us to rethink the entire way the web works. I’ll also briefly touch on many of the pitfalls that web developers still face since HTTP is stateless, and how various frameworks work to overcome this severe limitation of the web. And finally, I’ll briefly discuss alternative web technologies used to design browser based interfaces such as Flash, Silverlight, WPF/XAML and what I perceive as severe shortcomings of each of them.