WebApp Scaffolding: Yeoman, AngularJS and more

Continuing from what we had yesterday, I was still exploring other options on what framework we could use. After exploring CakePHP a little more, I finally decided that it might not be that suitable for our project after all.

Jon and I also wanted to start out with some UI design first, so I thought I should quickly create a workflow for doing the layouts. We chose to use Bootstrap as our front-end UI framework. Having tried GruntJS and Bower before, I was thinking of how to best scaffold the application. That was when I found Yeoman.

Yeoman is a scaffolding tool that helps you get started on webapps. It has a community-maintained list of generators that scaffolds a new application by pulling in relevant Grunt tasks and Bower dependencies that you might need for your build.

Out of the 720 community-maintained generators for Yeoman, I decided to give the angular-fullstack generator a try. Besides being the most popular generator, it also seemed to fit our needs. This generator generates a webapp built on AngularJS and comes with an Express server. It also comes with useful build tools that simplify development.

Most of the day was spent figuring out how that app is scaffolded. The hardest part was figuring how everything worked together. After spending some time learning the basics of AngularJS, I tried to modify the app to authenticate with NUS OpenID. For that, I gave Passport another try.

Spent the next few hours trying to get that working. Passport uses what they call ‘strategies’ to log users in to different sites. I ended up writing one to connect to NUS OpenID based on code from another OpenID strategy. Although it was just a few lines of modification, it took me quite awhile to get it working. I also published this ‘strategy’ as an npm package (passport-nus-openid) so that the webbapp can use it as a dependency.

The last few hours of the day also helped me better understand the model-view-controller architecture that AngularJS is based on.

Although Jon and I stay at different ends of the island, Google Hangouts was really useful in helping us to work together. Using screen sharing and voice chat, it was almost as good as meeting up physically to work on the project.

Setting Up the Development Environment

Started off the day setting up all the servers needed to get started on the Orbital project.

Our initial thoughts were to use CakePHP as a framework for the webapp, so I started by installing Apache, PHP and MySQL on my development machine.  Configuring PHP to work properly with Apache on Windows was a little more challenging than getting it working on Linux. I ended up having to grab some packages from ApacheLounge in order to get both to work together.

Since we wanted to use a RESTful API to interface with the database, I thought of giving NodeJS a try. Got started with a node application running on the ExpressJS framework.

Read and learnt more about npm, and also how to define project dependencies in the package.json file.

When Jon set up a git repository on GitHub for this, I was also thinking of how to separate config settings such as database credentials from code that will be committed. Read previously that this is a good practice as different developers working on the code may have different config settings depending on the development environment. Started off with a separate JS file to hold the constants but later found a more elegant solution – the config package. This package handles configuration files very well. It reads configuration files from the /config folder and determines which values to provide to the app.

Since we wanted also to log users in though OpenID, I looked around to see if there was any packages for that. Initially found and tried to implement passport-openid but ran into some problems. Decided to give the openid package a try instead, and it worked out pretty well.

I also grabbed phpMyAdmin and started modelling the tables based on what Jon and I discussed yesterday. After much thought, I ended up with the following database schema.

CREATE TABLE IF NOT EXISTS `poll` (
`id` int(11) NOT NULL,
 `owner_id` int(11) NOT NULL,
 `question` text NOT NULL,
 `type` int(11) NOT NULL,
 `status` int(11) NOT NULL,
 `created` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
 `modified` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00' ON UPDATE CURRENT_TIMESTAMP
) ENGINE=InnoDB DEFAULT CHARSET=utf8 AUTO_INCREMENT=1 ;

CREATE TABLE IF NOT EXISTS `poll_answers` (
`id` int(11) NOT NULL,
 `owner_id` int(11) NOT NULL,
 `question_id` int(11) NOT NULL,
 `answer` text NOT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8 AUTO_INCREMENT=1 ;

CREATE TABLE IF NOT EXISTS `poll_meta` (
`id` int(11) NOT NULL,
 `poll_id` int(11) NOT NULL,
 `meta_key` varchar(255) NOT NULL,
 `meta_value` text NOT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8 AUTO_INCREMENT=1 ;

CREATE TABLE IF NOT EXISTS `user` (
`id` int(11) NOT NULL,
 `openid` text NOT NULL,
 `created` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP
) ENGINE=InnoDB DEFAULT CHARSET=utf8 AUTO_INCREMENT=1 ;

ALTER TABLE `poll`
 ADD PRIMARY KEY (`id`);

ALTER TABLE `poll_answers`
 ADD PRIMARY KEY (`id`);

ALTER TABLE `poll_meta`
 ADD PRIMARY KEY (`id`);

ALTER TABLE `user`
 ADD PRIMARY KEY (`id`);

ALTER TABLE `poll`
MODIFY `id` int(11) NOT NULL AUTO_INCREMENT;
ALTER TABLE `poll_answers`
MODIFY `id` int(11) NOT NULL AUTO_INCREMENT;
ALTER TABLE `poll_meta`
MODIFY `id` int(11) NOT NULL AUTO_INCREMENT;
ALTER TABLE `user`
MODIFY `id` int(11) NOT NULL AUTO_INCREMENT;

While doing that, I found out that MongoDB allowed you to store data as collections of documents, and you could easily nest data within each document. I decided to give MongoDB a try. While I did have some past experience with relational databases, MongoDB was fundamentally different as it was non-relational. Before today, I did have some idea of NoSQL databases though I have never tried one before. Spent a few moments exploring it on try.mongodb.org, a MongoDB shell within the web browser, before installing it on my local machine to further learn about it.

At the end of the day, I’m still not sure if MySQL or MongoDB would be more suitable for this project. Also, I’m still considering if separating UI from logic completely though a REST API would be a good choice. I’ll probably start working on the UI first before coming back to these decisions.

Setting Expectations

For starters

Before embarking on the project, I think it was important for our team to set achievable goals and what we want to gain out of doing Orbital.

We’re both named Jonathan, so pedantically it is going to be interesting as we keep a shared log on our progress referring to each other as Jon, so do keep a look out for who the author is!

Our main focus is to attain the 4 credits that would contribute towards graduation, gaining valuable experience and technical know-how in the process as we explore into the technologies we would have to learn for us to be able to deploy a functioning web application.

We have decided on embarking on the intermediate level, the Project Gemini. To attain level of achievement, we’re required to meet the following requirements:

  • Attend the Liftoff workshop.
  • Log the time you spent on the programme, with a total time of at least 130 hours.
  • Participate in the end of summer showcase Splashdown.
  • Complete the peer-grading exercises.
  • Show progress on your project over all three evaluation milestones (late May, late June, late July).
  • Score an average of at least 2.5 stars (better than satisfactory) out of 4 on feedback given to other teams, and on own peer-graded project.
  • Complete at least 4 additional extension milestones.

Post-Liftoff

Over the 2-day introductory workshop, we were introduced to Python and Google App Engine and were briefly initiated to the development of a very basic web application. On top of them, we were also briefly introduced to HTML and CSS, as well as a basic Git coverage by Ng Zhi An.

Our project idea basically addresses polling (or getting feedback),  which professors and teaching assistants can use during lessons to gather responses for purposes such as to assess the general understanding of the audience, or to have a live quiz. Effectively it replaces the need for the clicker device which is costly to replace and students and teachers alike will not have to worry about the operational and logistical concerns of that particular device.

More importantly, the concepts of the Agile Methodology and Design Thinking Process that were introduced on the first day were crucial in helping us brainstorm and create a rough outline and sketch of the system we were envisioning. To go about this, Jon and I spent 3 hours that evening doing up sketches which can be found here.

Expectations

Having entirely no experience in web development, or any kind of application development at all, I am feeling rather ambitious in taking on this project with a partner with over 8 years of experience in dabbling with web technologies as they evolved through the early years of the millennium. Having said so, we are determined to accomplish the intermediate level of achievement for glory and something sweet to call our own.

Looking back on the freshmen year as Information Systems majors, we have finished the following relevant modules which hopefully are able to aid our understanding of things we are about to face:

  • CS1010 – Programming Methodology
  • CS1231 – Discrete Structures
  • CS2100 – Computer Organisation
  • CS1020 – Data Structures and Algorithms I
  • IS2102 – Requirements Analysis and Design
  • IS2101 – Technical Communication
  • IS1105 – Strategic IT Applications

Although I find them largely inapplicable in our application development, hopefully we are able to draw links and apply what has been learnt previously to build upon the knowledge that will help in future modules we will read in our time in NUS.

Challenges

  • Mismatch of skillsets

Jon is experienced and has a better understanding of databases and web development methodologies and best practices which I don’t. I aim to bridge the gaps in understanding as well as to pick up the nuggets of experience and knowledge between our differences. Developing the project together will be difficult because I might not be able to follow what Jon will be coding in the backend. What’s harder to do is to find time to get together since we live on opposite ends of the island so that we can go through the thought processes together.

  • Schedules

Jon leaves for Berkeley Summer School on 8th June. I’m currently on an internship at an IT consultancy startup. Taking on Orbital may have been ambitious on hindsight, but nonetheless I believe it to be a valuable motivating factor to pick up HTML, CSS, Javascript that would be valuable in redesigning websites that I can cross-apply to my projects at the firm.

We aim to finish a functioning model of our project by the time Jon leaves for Berkeley, and as such we may need to perform sprints and rapidly prototype our application. With such a short time-frame, it is going to be difficult for me to get up to speed to address our mismatch of skills and to contribute towards our 8th June target.

  • Deviation from the program

For the project, we decided that we will not be following the Orbital programme outline for Py/GAE since we have decided that it is not suitable for what we want to build.

Node.js/Express.js framework, websockets, and a solid database requires fundamental understanding of networks and databases which I am unfamiliar with. Developing the front-end also requires me to get on with the Bootstrap programme and pick up web design best practices for HTML and CSS. These frameworks come with their own set of dependencies which may need some familiarisation as well.

Essentially these are the things we are doing as milestone extensions from the basic Vostok level – which means I’m facing a very steep learning curve on terra incognito.

How now brown cow?

To resolve the issue of communication, we’ll attempt to dedicate at least 2 hours each day to do a Google Hangout for Jon to explain what is being done while I bulldoze through my curriculum of the various technologies.

We are trying Nitrous.io on top of our Github repository for collaborative code editing since there is support for Node.js and it seems like a pretty cool IDE for Pair Programming. We’ll see how that works out. (edit 16/05/14: nitros.io: Looks like your box was shutdown due to inactivity. If you want your boxes to never shutdown, please upgrade your plan; Oh my goodness.) 

I am coming up with a roadmap to establish realistic dates for the application development to help give us more visibility on our progress.

For subsequent updates, we’ll be blogging to track the following things to aid our learning and reflections:

  • Takeaways for the Day
  • Hours spent on respective items (possible extension: productivity improvement)
  • Problems encountered and our workarounds/solutions

We’ll adopt a progressive approach to developing the project and explore breadth first before delving deep into each domain to get a good grasp of what is required in our application.