September 7th, 2008
6:16 pm
Python
Python 2.6 will add a multiprocessing module that supports fork/join semantics and management of process pools. However even in this multiprocessor environment it is still necesary to do some asynchronous. The main way that processes communicate in this environment are through shared queues (implemented as socket-pairs or unix pipes under the hood), but we need a method for the UI to be notified about updates to any queues it is listening on without blocking (for example in a UI). Luckily it is possible to get at underlying file-descriptor, which can then be managed with select() or some abstraction thereof. Unfortunately there's no official support for this but we can still poke around inside the implementation class to get at the details. The code below shows how to do this with GTK. This just spawns off a sub-process that counts up and sends the current number to the UI via a queue, which then displays the number:
PYTHON:
-
from __future__ import print_function
-
import gtk, gobject, time, multiprocessing, Queue
-
-
-
class UI:
-
-
def destroy(self, widget, data=None):
-
self.outq.put("quit")
-
gtk.main_quit()
-
-
def __init__(self):
-
self.window = gtk.Window(gtk.WINDOW_TOPLEVEL)
-
self.window.connect("destroy", self.destroy)
-
self.window.set_border_width(10)
-
-
self.label = gtk.Label("Not Set!")
-
self.window.add(self.label)
-
-
self.label.show()
-
self.window.show()
-
-
def update(self, fd, cond):
-
val = self.inq.get()
-
self.label.set_text("Value is %s" % val)
-
return True
-
-
def main(self, outq, inq):
-
self.outq = outq
-
self.inq = inq
-
-
# This is the tricky bit, we get the socket FD of the
-
# underlying pipe from the queue object. We hope that the
-
# internal reader variable doesn't change.
-
fd = inq._reader.fileno()
-
-
# This tells GTK to watch this FD and notify us when data is
-
# available. We use newer API, gdk_input_read is deprecated
-
gobject.io_add_watch(fd, gobject.IO_IN, self.update)
-
gtk.main()
-
-
-
def counter(inq, outq, interval):
-
count = 1
-
again = True
-
while again:
-
print("Putting %s on queue" % count)
-
outq.put(count)
-
count += 1
-
-
try:
-
print("Reading from queue...")
-
ret = inq.get(True, interval)
-
if ret == "quit":
-
print("Got quit message")
-
again = False
-
except Queue.Empty:
-
# Timed-out, carry on
-
pass
-
-
-
if __name__ == "__main__":
-
q1 = multiprocessing.Queue()
-
q2 = multiprocessing.Queue()
-
-
proc = multiprocessing.Process(target=counter, args=(q1, q2, 1,))
-
proc.start()
-
-
ui = UI()
-
ui.main(q1, q2)
-
-
proc.join()
Obviously if the underlying implementation variables change this will break; hopefully a more standard way of getting at this information will be added to the API.
July 8th, 2008
9:05 pm
Misc
I've been importing my old mail into Gmail and discovering some interesting snippets I'd all but forgotten. The following is from a mail I sent a friend when I was the lead programmer for a startup in 2000 doing mortgage brokering online:
One interesting event was that we appeared (for free) on a current-affairs program in various cities one night. [Note: It was "Today Tonight".] That was mental. They saw one of our press-releases about the "true cost of your mortgage", and thought it would make a good piece. We basically wrote the thing for them and supplied all the people that told their 'story' on it. Lesson: when you see an 'expose', ask who stands to benefit from it. Worked for us though. After it aired somewhere between 4000-8000 people logged onto the site *at the same time*.
Caveat Spectator.
May 18th, 2008
9:03 pm
Misc
My old team-leader Scott "Satan" MacGibbon pointed out a couple of Economist articles that touch on some of the things I've been thinking about...
The first article is on the migration to less centralised and specialised workplaces. Most of it is another variation on the "Starbucks is the new cubicle" theme that seems to be discovered by journalists every couple of months, but I found the discussion on the revival of "Third Places" interesting, not least because I hadn't actually encountered the term before.
The other is a short and (for the Economist) to-the-point article on the further damage that the boomer retirement wave of housing sales may do to the already seriously-wounded US property market. Most of it is along the line of the points I made below, in that the same logic applies in the Australian market, maybe even more so given that the investment property has been the preferred method of salting something away for the future for some time now. However his paragraph caught my eye:
Suburbs, which swelled with the baby-boomers, may begin to decline. If the building industry contracts, home prices may remain more stable. Or developers may switch to serving the old, building more compact housing near amenities. Towns may make new efforts to attract immigrants, who already accounted for 40% of the growth in homeownership between 2000 and 2006.
Unfortunately what the article fails to address is how this will interact with the rising cost of private transportation caused by peak-oil (or the less hysterical "plateau-oil", as John Quiggin terms it). My first thought is that there are two ways this can play out; rising costs will further speed the collapse of the suburban lifestyle, or they will force the accommodation of more flexible work conditions. But there is another factor; the most remarkable effect of the changes in the oil market is likely to be in food production. There is already a recognition that most of the food we eat travels are ridiculous distance to our plates (supposedly ~2,500Km on average, although I can't find a source for that). This may cause a fundamental change in the way we live as it becomes more economical to move closer to the centres of food production. This in turn may revitalise the currently bland and homogeneous environment of the suburban tracts; it's possible that in the end the suburbs may become the country villages of the 21st century.
May 8th, 2008
2:45 pm
Misc
Last November I posted the following to a private mailing-list in a discussion on telecommuting:
My feeling is that such arrangements will become more common in the future. The reasoning behind this is that as Australia's population ages there will be an increasing shortage of skilled knowledge workers, and consequently working conditions will become more negotiable. Add to this the likely gradual increase in oil prices (reflected at the pump) and the continuing high cost of property near the city and this option will be attractive to many employees.
If the above holds (and I may very well be wrong) then this will start to really kick-in in 2010 (when the baby-boomers start to retire), peaking around 2015. But we're already starting to see some of the effects, with companies going to lengths to hire and retain younger employees ....
The Australian Financial Review today had a article (paywalled so no link) on the coming hiring crisis that reflects a lot of what I've said there. Particularly interesting is this quote from the Terence Budge of the Australian Institute of Company Directors:
This generation want to live where they work and not just be in the office from nine to five ... They want to do some work over a coffee at the cafe, something else on their laptop at night ...
I think this problem is going to see a lot more attention over the next couple of years, and is going to have other side-effects that are as yet unseen. Certainly it is going to affect the tools that are used by these increasingly mobile workers; SaaS, wikis and other manifestation of the internet-as-the-platform are likely to have an even greater uptake than we are already seeing.
Another quote that caught my eye is this from Craig Perrett of act3:
Mr Perrett said only about 25 per cent of workers at retirement age believed they could afford to retire and many were rejecting the traditional notion of retirement.
If it is true that 75% can't afford to retire then this is going to have some huge implications. The first one that springs to mind is that unless these underfunded retiree's are prepared to work full time they are going to have to find money from somewhere, and for most the only significant asset they have is their home. This will lead to a glut of housing entering the market in the medium term. Ultimately this and the decentralisation of the workplace could have a far greater effect on the Sydney housing crisis than government intervention.
January 15th, 2008
3:18 pm
Misc
So today, after a mad 24-hour deadline to deliver a bank-cheque (not helped by a crashed bank network) myself and Maree officially took ownership of our first home:
Ya ain't in Newtown anymore, boy!
We've bought up in Katoomba, for two main reasons:
- This is where we eventually want to live
- This is where we could afford
Lets call it one of life's fortuitous coincidences that these two things came together.
The first reason probably doesn't need much explanation; the Blue Mountains is a World Heritage listed area, recognised for its beauty and culture, and last year became recognised as a Cittaslow area. Katoomba itself is a small but vibrant town with many of the benefits of Newtown, not least the cafes and other eateries. Given I have a long history of burning myself out at work I had promised Maree (and myself) that I would work towards getting a better work/life balance and this is a key part of that plan (the move from military projects to the more healthy environments of Vislab and Atlassian were the first steps in that direction).
The financial aspect should be fairly obvious to anyone who's lived in Sydney in recent years; it's just too expensive to buy unless you're prepared to go into deep debt for a long time. Given I have a deep life-long aversion to debt this wasn't really an option for us. Although the price difference between Sydney and Katoomba isn't as big as it has been in the past, the few-hundred thousand saved can make a huge difference to the length of the loan; this is because the initial deposit will take up a larger proportion of the total cost (assuming you've actually been saving of course), meaning you get to the tipping point much faster:
What we'd be paying if we followed the rules (red being interest)
Add to this that I'm playing some games with full-offset accounts and we have a best-case scenario of completely owning the property in under 4 years*. Of course, in Katoomba you also get a hell of a lot more for your money; 1000m² at the end of a tree-lined street 5 minutes from the station in our case.
A more interesting question (at least to me) is "Why now?". The recent over-heated market aside, property rarely out-performs the shares market so logically we should invest elsewhere for now and then purchase nearer the time. The reasoning goes like this:
Australia is getting older. This is an accepted fact now, and is of some concern to economists and politicians (the latter of which are starting to realise what a giant ponzi-scheme has been going on for decades). This is going to come to a head in the next few years, specifically 2010-2015 (the period when the baby-boomer generation start retiring**). One of the side-effects of this is that a large number of people currently living in the city and suburbs are going to start migrating out to the sea-change and tree-change areas. This in turn will drive up the prices, possibly to the level of the city (if this seems excessive check out the property prices in Leura some time). (There's a related effect to do with pay and work conditions for those left-behind in the workforce, but that's probably wandering too far off topic; Ross Gittins has written about this in his book.)
Given all this it seemed prudent to buy now rather than later before the market heats up.
Of course, it goes without saying that I could be completely fucking wrong about all of this.
The bigger question now is how to manage the distance to the city. Some people at Atlassian do commute from the mountains but I'm not sure we can handle that. Longer term I would like to come to an arrangement where I can work from home most of the time, but in the interim we're going to try it for six-months or so while keeping a rental place in the city; if it we can't keep it up we'll have to rent the house out for a few years before making the final move.
So all that remains now is to start moving our stuff up to the new place and get settled in. There's also some work and minor improvements to be done over the next couple of months so that will be taking up a lot of my weekends. All-in-all I have an odd feeling of adulthood about me, which is probably about 10 years late now.
* There's a strong case to be made that paying-off your mortgage quickly is not always the best option. One advantage of using a full-offset account to rapidly pay off the mortgage is that the principal is still available to us, so we can change this tactic later if we wish.
** I'm working on the assumption that the 1968 "Summer of Love" was the peak of the baby-boomer's coming-of-age (i.e. turning 18-21); some believe we'll start seeing the effects even sooner.
October 7th, 2007
4:09 pm
Misc
Back when I worked at Vislab I wrote a quick and dirty script that generated an RSS feed from the University of Sydney jobs page, mainly to suggest jobs to my girlfriend (who at that time worked in the Faculty of Arts and was looking for alternatives). Since I left bit-rot had set in and it stopped updating; I didn't care, both of us had moved on to other things.
However the other day I suggested to a friend they look at University jobs and realised I should probably reinstate the feeds. The problem is that University jobs are generally not advertised on boards such as Seek and have the be checked manually, and the lack of feeds makes this situation worse. As a bonus I've also added a basic feed for UWS as that's what the original conversation was about; I may add more later, time permitting. The current crop of feeds can be found in this directory, or use the links below:
October 2nd, 2007
1:36 pm
Lisp
Some time ago I wrote a single-threaded server implementation in SBCL using its serve-event abstraction. However recently I've been working with ECL, which has some interesting possibilities due to its small size. However currently there is no abstraction of non-blocking IO, so I spent a bit of time porting the SBCL/CMUCL serve-event across. The hardest part about this was working out how to declare opaque C-structs with ECL lisp blocks in a robust way. In the end the old stalwart unsigned-char that saved the day, a technique I've documented elsewhere. But without further ado, the code:
(Update: This is now in ECL trunk; you can view an updated version here)
Lisp:
-
(defpackage "serve-event"
-
(:use "CL" "FFI" "UFFI"))
-
(in-package "serve-event")
-
-
(clines "#include <sys/select.h>")
-
-
(defstruct (handler
-
(:constructor make-handler (direction descriptor function))
-
(:copier nil))
-
;; Reading or writing...
-
(direction nil :type (member :input :output))
-
;; File descriptor this handler is tied to.
-
;; FIXME: Should be based on FD_SETSIZE
-
(descriptor 0)
-
;; Function to call.
-
(function nil :type function)
-
;; T if this descriptor is bogus.
-
bogus)
-
-
-
(defvar *descriptor-handlers* nil
-
#!+sb-doc
-
"List of all the currently active handlers for file descriptors")
-
-
-
;;; Add a new handler to *descriptor-handlers*.
-
(defun add-fd-handler (fd direction function)
-
"Arrange to call FUNCTION whenever FD is usable. DIRECTION should be
-
either :INPUT or :OUTPUT. The value returned should be passed to
-
SYSTEM:REMOVE-FD-HANDLER when it is no longer needed."
-
(unless (member direction '(:input :output))
-
;; FIXME: should be TYPE-ERROR?
-
(error "Invalid direction ~S, must be either :INPUT or :OUTPUT" direction))
-
(let ((handler (make-handler direction fd function)))
-
(push handler *descriptor-handlers*)
-
handler))
-
-
;;; Remove an old handler from *descriptor-handlers*.
-
(defun remove-fd-handler (handler)
-
#!+sb-doc
-
"Removes HANDLER from the list of active handlers."
-
(setf *descriptor-handlers*
-
(delete handler *descriptor-handlers*)))
-
-
;;; Add the handler to *descriptor-handlers* for the duration of BODY.
-
(defmacro with-fd-handler ((fd direction function) &rest body)
-
"Establish a handler with SYSTEM:ADD-FD-HANDLER for the duration of BODY.
-
DIRECTION should be either :INPUT or :OUTPUT, FD is the file descriptor to
-
use, and FUNCTION is the function to call whenever FD is usable."
-
(let ((handler (gensym)))
-
`(let (,handler)
-
(unwind-protect
-
(progn
-
(setf ,handler (add-fd-handler ,fd ,direction ,function))
-
,@body)
-
(when ,handler
-
(remove-fd-handler ,handler))))))
-
-
-
(defmacro fd-zero(fdset)
-
`(c-inline (,fdset) (:object) :void
-
"FD_ZERO((fd_set*)#0->foreign.data)"
-
:one-liner t
-
:side-effects t))
-
-
(defmacro fd-set (fd fdset)
-
`(c-inline (,fd ,fdset) (:int :object) :void
-
"FD_SET(#0, (fd_set*)#1->foreign.data);"
-
:one-liner t
-
:side-effects t))
-
-
(defmacro fd-isset (fd fdset)
-
`(c-inline (,fd ,fdset) (:int :object) :int
-
"FD_ISSET(#0, (fd_set*)#1->foreign.data)"
-
:one-liner t
-
:side-effects t))
-
-
(defun fdset-size ()
-
(c-inline () () :int "sizeof(fd_set)" :one-liner t :side-effects nil))
-
-
-
(defun serve-event (&optional (seconds 0))
-
"Receive pending events on all FD-STREAMS and dispatch to the appropriate
-
handler functions. If timeout is specified, server will wait the specified
-
time (in seconds) and then return, otherwise it will wait until something
-
happens. Server returns T if something happened and NIL otherwise. Timeout
-
0 means polling without waiting."
-
-
;; fd_set is an opaque typedef, so we can't declare it locally.
-
;; However we can fine out its size and allocate a char array of
-
;; the same size which can be used in its place.
-
(let ((fsize (fdset-size)))
-
(with-foreign-objects ((rfd `(:array :unsigned-char ,fsize))
-
(wfd `(:array :unsigned-char ,fsize)))
-
(fd-zero rfd)
-
(fd-zero wfd)
-
-
(let ((maxfd 0))
-
;; Load the descriptors into the relevant set
-
(dolist (handler *descriptor-handlers*)
-
(let ((fd (handler-descriptor handler)))
-
(ecase (handler-direction handler)
-
(:input (fd-set fd rfd))
-
(:output (fd-set fd wfd)))
-
(when (> fd maxfd)
-
(setf maxfd fd))))
-
-
(let ((retval
-
(c-inline (rfd wfd (1+ maxfd) seconds)
-
(:object :object :int :int) :int
-
"{ struct timeval tv;
-
tv.tv_sec = #3;
-
tv.tv_usec = 0;
-
@(return) = select(#2, #0->foreign.data,
-
#1->foreign.data,
-
NULL, &tv); }"
-
:one-liner nil
-
:side-effects t)))
-
(cond ((zerop retval) nil)
-
((minusp retval)
-
(error "Error during select"))
-
(t
-
(dolist (handler *descriptor-handlers*)
-
(let ((fd (handler-descriptor handler)))
-
(if (plusp (ecase (handler-direction handler)
-
(:input (fd-isset fd rfd))
-
(:output (fd-isset fd wfd))))
-
(funcall (handler-function handler)
-
(handler-descriptor handler))))))))))))
-
-
-
;;; Wait for up to timeout seconds for an event to happen. Make sure all
-
;;; pending events are processed before returning.
-
(defun serve-all-events (&optional (timeout 0))
-
"SERVE-ALL-EVENTS calls SERVE-EVENT with the specified timeout. If
-
SERVE-EVENT does something (returns T) it loops over SERVE-EVENT with a
-
timeout of 0 until there are no more events to serve. SERVE-ALL-EVENTS returns
-
T if SERVE-EVENT did something and NIL if not."
-
(do ((res nil)
-
(sval (serve-event timeout) (serve-event 0)))
-
((null sval) res)
-
(setq res t)))
-
-
-
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
-
;; Test Example
-
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
-
-
;; (defun test-stdin ()
-
;; (format t "DOING STDIN~%")
-
;; (with-fd-handler (0 :input #'(lambda (fd) (declare (ignore fd))
-
;; (format t "Got data~%")
-
;; (read-char)))
-
;; (loop ;; FIXME: End condition
-
;; (format t "Entering serve-all-events...~%")(force-output)
-
;; (serve-all-events 5)
-
;; (format t "Events served~%"))))
I've submitted a patch to the ECL list, and there's been some interest in integrating it so it should be part of it soon.
Update: This was added in the ECL 0.9j release.
October 1st, 2007
1:21 pm
Misc
Jamie Zawinski has just posted some instructions on a basic backup scheme. While the general advice is good, there are a couple of improvements that could be made, so I thought I'd describe my home backup system.
The main problem the the rsync script Jamie posted is that if you accidentally delete some important files the next time your backup runs rsync will delete the backups of those files. This is due to the --delete command. The obvious solution is to remove that option, but that will tend to leave a very messy filesystem over time. My preferred solution is to use the --backup and --backup-dir options to create a simple form of incremental backup. My script looks like this:
Bash Script:
-
dir=/media/backup
-
stamp=`date +'%Y%m%d'`
-
bdir=$dir/.incremental.$stamp
-
-
rsync -Pav --backup --backup-dir=$bdir --delete-after \
-
/home /etc /usr/local /opt \
-
--exclude-from=/home/ssmith/.backup-exclude \
-
$dir
Note that my script doesn't do a full-drive image like Jamie's as I don't find that important. Also, my script excludes some files I don't care about (i.e. can replace easily). Regardless, the principle is the same.
However this doesn't cover the house-burns-down scenario. Obviously the above script could easily be used with a drive you then bring to work as with JWZ's suggestion. But in my experience you will simply never do that.
The solution? I use a near-identical script to backup to a remote host. If you already have a machine somewhere you can use that, or there are many remote backup services, some of which have a direct rsync service. While the initial upload will be pretty large the subsequent incremental uploads will be much quicker, especially if you have ADSL2+, which has 1Mbit uploads. The cool part is that this leverages the uploads-are-free aspect of most Australian ISPs.
The only issue with this is that your data is possibly exposed to third-parties. My normal solution to this is to place any critical data in a directory that is encrypted using encfs. However I'm now having a look at Duplicity as an alternative, which works in much the same way but has GnuPG signing and encryption. It also supports more protocols, such as webdav and Amazon's S3 service.
November 26th, 2006
11:30 am
Misc
At Atlassian a few of us are growing mustaches for Movember; mine's a bit crap but some others are managing to get some impressive porno 'taches going. Personally I don't think men's health issues get enough attention so for me it's largely a solidarity/awareness thing rather than actively soliciting for sponsors. But other than not shaving one part of my face I hadn't actually been thinking about it too much.
However yesterday I went to a friend's barbecue. Nothing remarkable in itself; beer, fun and mountains of snags were had by all. But as we were leaving a girl who I'd only vaguely been aware of and hadn't really talked to ran up and asked if I was growing a mustache for Movember. I said 'yes' and she suddenly expressed a surprising amount of emotion. It turns out that she had lost both her husband and father to cancer, and was really grateful that was helping to raise awareness of the issue, and that people were finally starting to give it some attention.
This doesn't really change anything of course, but it's funny how what to me is a bit of a lark with a point behind it can mean a fantastic amount to someone else.
Update: By popular request ...
They even made some custom beer-bottles for the last day:
October 22nd, 2006
12:14 pm
Misc
This may be the funniest (and spot-on) thing I've seen a while:
The scene: A room. Two men sit across a table from each other. One is a representative of the GRID. The other is a member of a funding council.
GRID: Condor Grid eScience jihad. Accessgrid! Lustre! 3d animation! Inter-disciplinary research! derka derka middleware layer!
Research council: Grid! Grid! Grid!
(MONEY falls from the sky. Its origins are never adequately explained)
In a previous life I was the "AccessGrid Project Manager" at the University of Sydney, and ended up engineering an alternative to the AccessGrid protocols based on open standards rather than the grid infrastructure (which really has nothing to do with the AccessGrid at all). This was ultimately fruitless and the above pretty-much sums-up why I was always on a hiding to nothing with that idea.