Posted by Ian Holsman
Hey GOOG.. are you listening..
on your driver route map instead of showing the ‘fastest’ way, how about the ‘safest’ way?
I would like to see a route which has the least possible amount of ‘black spots’.
It sounds like a perfect idea for a mashup, but driving directions isn’t exportable (I believe). It would be cool if google had a API which would show you 3-4 possible routes, and then allow you to add stuff like this.
Posted by Ian Holsman
I was at a conference yesterday, and one of the speakers was complaining about how the documentation his company produced was useless after it was written.
I’m not going to debate this, I’ve never been a fan of large wads of paper. but while he was talking I thought of another use of documentation. It isn’t the document itself that is important, but the act of creating it. (so sorry agile guys.. I still believe documentation is important and needs to be written).
If your document template is good, it should be asking you probing questions, and force you to think about what your trying to achieve. Be it a business plan, a architecture document, or a test plan.
The act of reading the different sections, talking to people about them, and putting those thoughts to paper is 90% of what the document is about, not that the documentation will be read of used by other people after this process has finished.
The important consequence of this thought involves who you get to participate in the document creation. You need to involve the people who traditionally were the recipients (the people on the other side of the wall catching the thing you threw over) of the document in the creation of it.
When it comes time to actually implementing the document, they use it to remind themselves of the conversations and thinking they had when they wrote it. The other benefit will be grounding those founders/architects/ivory tower people with some reality.
Posted by Ian Holsman
Recently I’ve been thinking about virtualization.
Initially I thought it was a slam dunk. it would make it nice and easy to maintain a system which consists of multiple load balanced tiers/applications…
if you need to increase the capacity of a given tier you would simply install the image on another box, and away you go, checkpointing would allow you to recover from a crash and just move the images to another machine.. it sounds like nirvana.
but I actually thought about it a bit more. The first thought which came to me was OS license costs. Right now the ‘standard’ practice where we are is to run multiple applications on a machine, and share the OS. .. but with virtualization you wouldn’t do this, and would then need 1 license per virtual instance… which might make it more expensive. (I haven’t checked the licensing agreements, but I don’t think they have a clause about virtual OS in there yet)
what virtualization means to me is more logical machines. and that in turn brings other management headaches.
how do you monitor them?
all of a suddenly previous uncorrelated applications can interact with each. The standard “it’s slow” that you get from a user can now mean that the slowness is not being caused by the application components anymore, not to mention the sheer number of extra data points you will need to look at and collect (as each logical OS will have it’s own CPU and memory stats).
do you care about the checkpointing?
most of the transactions I see are OLTP. they last for a couple of seconds, and if the machine goes away while we are processing we don’t care… we ensure we are always in a consistent state at other layers, and the user will have to re-login (it’s painful, but hardware doesn’t break down too often). so this checkpointing feature doesn’t really give me anything.
Instant capacity?
The other neat thing I hear is about how easy it is to just add a instance when a tier gets overused. I’m not sure if this is a practical thing either. In most cases we have a hardware load balancer which separates the tiers of a application, and we avoiding changing them as much as possible. (network load balancers seem to be the most unstable/fragile pieces of equipment in my experience).. so adding extra instances is not going to be that easy.
provisioning and configuration.
While I have seen some pretty good procedures to do provisioning at CNET, it was by no means perfect, or completely automated. when push came to shove, we could get a new machine up and running in a couple of hours… I don’t see this changing with virtual appliances.
so while I think virtualization is a neat idea, i think there things you need to get in place before you even think about it. Provisioning, load balancing, and monitoring for a start.
Posted by Ian Holsman
Brian Aker starts work on a memcache engine for mysql. so your memcache cache acts just like a table.
the big thing here which I’ve seen asked for a couple of times on the memcached list is the ability to see a list of keys.
mysql > select * from foo1 WHERE k=”mine”;
freaking amazing.. I love these kind of mashups.
and the 2nd important event.
Django is starting a branch to integrate SQLAlchemy
Posted by Ian Holsman
or at least I hope too..
I’ve just put a proposal into the Apache Incubator for Fluoride
The aim of this proposal is to create a server which can track statistics on who is reading your feeds, and what they are reading and integrate these statistics back into your central log processing architecture.
The intended audience is for publishers who have lots of feeds and need to get decent statistics back into their data wharehouses.
some people have been calling this a open source feedburner… I don’t think it is.. the aim of the proposal is to create a piece of software to run on your own machines, not a hosted service.
It’s at the idea stage at the moment, and I’m trying to attract other developers who are interested.
so if your interested speak up!
Posted by Ian Holsman
I submitted this to the django core a while back, and they weren’t interested…
So i have put it in my open SVN repo for others to use if they want to.
it’s called ‘run_jobs’.
what you do is in your [appname].management.py file you define a set of cleanup tasks that your application needs run on a regular basis.
for example.. here are two which clean up the session and cache tables
def session_cleanup():
cursor = connection.cursor()
cursor.execute("DELETE FROM %s WHERE %s < NOW()" % \
(backend.quote_name('django_session'), backend.quote_name('expire_date')))
transaction.commit_unless_managed()
def cache_cleanup():
from django.conf import settings
import os
if settings.CACHE_BACKEND.startswith('db://'):
table_name = settings.CACHE_BACKEND[5:]
cursor = connection.cursor()
cursor.execute("DELETE FROM %s WHERE %s < UTC_TIMESTAMP()" % \
(backend.quote_name(table_name), backend.quote_name('expires')))
transaction.commit_unless_managed()
dispatcher.connect(session_cleanup, signal=signals.run_daily_jobs)
dispatcher.connect(cache_cleanup, signal=signals.run_daily_jobs)
15 9 * * * /usr/local/src/magik/zilbo/bin/run_jobs.py --settings=zilbo.settings runjobs daily
15 7 * * * /usr/local/src/magik/zilbo/bin/run_jobs.py --settings=zilbo.site2 runjobs daily
for the 2 sites I have on that machine. which would fire off once a day, and then run the session and cache clean up jobs.
so far .. pretty mundane.
now lets say I wrote another application called ‘counter’ which counts how often people view a given object. I really only want to keep stats for 8-10 weeks.. so now all I need to do is create a counter.management.py file and add the write a similar function and connect it to the signal I care about (daily in this case as well).
now.. the next time I run my batch job.. my counter objects will get cleaned as well.
without having to remember to edit the crontab.
Posted by Ian Holsman
on a similar vein to Coulix’s contribution, I have written a context_processor which allow you to see what city/region/etc your visitor is coming in from
It uses MaxMind’s GeoIP database.
warning.. this module is GPL due to MaxMind’s code being GPL. putting it into your codebase will make it catch a virus.
Posted by Ian Holsman
So.. After doing my screencast covering how easy it was to get a django site up and running on webfaction I thought it would be only fair to show people how easy it was to get a rails app (typo) up and running on webfaction as well
Thanks to Remi for letting me do this.
Posted by Ian Holsman
on some shared hosting environments like webfaction they install a reverse proxy solution where you run a apache2 webserver on a custom port, and the main webserver forwards the request through to you.
This works great, and I’ve been using things like this for years, but the problem you get is that all the requests appear to be from 127.0.0.1 (or the machine which is doing the proxying for you) which isn’t the best if you want to record that stuff.
webfaction fixes this in the log files for you automatically, but that doesn’t help you if you want to do something with the IP# in django.
enter FixIP it takes the X-Forwarded-For header sent by compliant servers which holds the real IP and makes django work properly.
installation is easy.. just add the routine to your middleware settings ala
MIDDLEWARE_CLASSES = (
"django.middleware.common.CommonMiddleware",
"zilbo.common.utils.middleware.fixip.FixIP",
...