Usyd positions feed updated

Back when I worked at Vislab I wrote a quick and dirty script that generated an RSS feed from the University of Sydney jobs page, mainly to suggest jobs to my girlfriend (who at that time worked in the Faculty of Arts and was looking for alternatives). Since I left bit-rot had set in and it stopped updating; I didn’t care, both of us had moved on to other things.

However the other day I suggested to a friend they look at University jobs and realised I should probably reinstate the feeds. The problem is that University jobs are generally not advertised on boards such as Seek and have the be checked manually, and the lack of feeds makes this situation worse. As a bonus I’ve also added a basic feed for UWS as that’s what the original conversation was about; I may add more later, time permitting. The current crop of feeds can be found in this directory, or use the links below:

More weekend lisp hacking: serve-event for ECL

Some time ago I wrote a single-threaded server implementation in SBCL using its serve-event abstraction. However recently I've been working with ECL, which has some interesting possibilities due to its small size. However currently there is no abstraction of non-blocking IO, so I spent a bit of time porting the SBCL/CMUCL serve-event across. The hardest part about this was working out how to declare opaque C-structs with ECL lisp blocks in a robust way. In the end the old stalwart unsigned-char that saved the day, a technique I've documented elsewhere. But without further ado, the code:

(Update: This is now in ECL trunk; you can view an updated version here)

Lisp:
  1. (defpackage "serve-event"
  2.   (:use "CL" "FFI" "UFFI"))
  3. (in-package "serve-event")
  4.  
  5. (clines "#include <sys/select.h>")
  6.  
  7. (defstruct (handler
  8.             (:constructor make-handler (direction descriptor function))
  9.             (:copier nil))
  10.   ;; Reading or writing...
  11.   (direction nil :type (member :input :output))
  12.   ;; File descriptor this handler is tied to.
  13.   ;; FIXME: Should be based on FD_SETSIZE
  14.   (descriptor 0)
  15.   ;; Function to call.
  16.   (function nil :type function)
  17.   ;; T if this descriptor is bogus.
  18.   bogus)
  19.  
  20.  
  21. (defvar *descriptor-handlers* nil
  22.   #!+sb-doc
  23.   "List of all the currently active handlers for file descriptors")
  24.  
  25.  
  26. ;;; Add a new handler to *descriptor-handlers*.
  27. (defun add-fd-handler (fd direction function)
  28.   "Arrange to call FUNCTION whenever FD is usable. DIRECTION should be
  29.   either :INPUT or :OUTPUT. The value returned should be passed to
  30.   SYSTEM:REMOVE-FD-HANDLER when it is no longer needed."
  31.   (unless (member direction '(:input :output))
  32.     ;; FIXME: should be TYPE-ERROR?
  33.     (error "Invalid direction ~S, must be either :INPUT or :OUTPUT" direction))
  34.   (let ((handler (make-handler direction fd function)))
  35.     (push handler *descriptor-handlers*)
  36.     handler))
  37.  
  38. ;;; Remove an old handler from *descriptor-handlers*.
  39. (defun remove-fd-handler (handler)
  40.   #!+sb-doc
  41.   "Removes HANDLER from the list of active handlers."
  42.   (setf *descriptor-handlers*
  43.         (delete handler *descriptor-handlers*)))
  44.  
  45. ;;; Add the handler to *descriptor-handlers* for the duration of BODY.
  46. (defmacro with-fd-handler ((fd direction function) &rest body)
  47.   "Establish a handler with SYSTEM:ADD-FD-HANDLER for the duration of BODY.
  48.    DIRECTION should be either :INPUT or :OUTPUT, FD is the file descriptor to
  49.    use, and FUNCTION is the function to call whenever FD is usable."
  50.   (let ((handler (gensym)))
  51.     `(let (,handler)
  52.        (unwind-protect
  53.            (progn
  54.              (setf ,handler (add-fd-handler ,fd ,direction ,function))
  55.              ,@body)
  56.          (when ,handler
  57.            (remove-fd-handler ,handler))))))
  58.  
  59.  
  60. (defmacro fd-zero(fdset)
  61.   `(c-inline (,fdset) (:object) :void
  62.              "FD_ZERO((fd_set*)#0->foreign.data)"
  63.              :one-liner t
  64.              :side-effects t))
  65.  
  66. (defmacro fd-set (fd fdset)
  67.   `(c-inline (,fd ,fdset) (:int :object) :void
  68.              "FD_SET(#0, (fd_set*)#1->foreign.data);"
  69.              :one-liner t
  70.              :side-effects t))
  71.  
  72. (defmacro fd-isset (fd fdset)
  73.   `(c-inline (,fd ,fdset) (:int :object) :int
  74.              "FD_ISSET(#0, (fd_set*)#1->foreign.data)"
  75.              :one-liner t
  76.              :side-effects t))
  77.  
  78. (defun fdset-size ()
  79.   (c-inline () () :int "sizeof(fd_set)" :one-liner t :side-effects nil))
  80.  
  81.  
  82. (defun serve-event (&optional (seconds 0))
  83.   "Receive pending events on all FD-STREAMS and dispatch to the appropriate
  84.    handler functions. If timeout is specified, server will wait the specified
  85.    time (in seconds) and then return, otherwise it will wait until something
  86.    happens. Server returns T if something happened and NIL otherwise. Timeout
  87.    0 means polling without waiting."
  88.  
  89.   ;; fd_set is an opaque typedef, so we can't declare it locally.
  90.   ;; However we can fine out its size and allocate a char array of
  91.   ;; the same size which can be used in its place.
  92.   (let ((fsize (fdset-size)))
  93.     (with-foreign-objects ((rfd `(:array :unsigned-char ,fsize))
  94.                            (wfd `(:array :unsigned-char ,fsize)))
  95.       (fd-zero rfd)
  96.       (fd-zero wfd)
  97.  
  98.       (let ((maxfd 0))
  99.         ;; Load the descriptors into the relevant set
  100.         (dolist (handler *descriptor-handlers*)
  101.           (let ((fd (handler-descriptor handler)))
  102.             (ecase (handler-direction handler)
  103.               (:input (fd-set fd rfd))
  104.               (:output (fd-set fd wfd)))
  105.             (when (> fd maxfd)
  106.           (setf maxfd fd))))
  107.  
  108.         (let ((retval
  109.                (c-inline (rfd      wfd    (1+ maxfd) seconds)
  110.                          (:object :object :int       :int) :int
  111.                          "{ struct timeval tv;
  112.                             tv.tv_sec = #3;
  113.                             tv.tv_usec = 0;
  114.                             @(return) = select(#2, #0->foreign.data,
  115.                                                    #1->foreign.data,
  116.                                                    NULL, &tv); }"
  117.                   :one-liner nil
  118.                   :side-effects t)))
  119.       (cond ((zerop retval) nil)
  120.             ((minusp retval)
  121.              (error "Error during select"))
  122.             (t
  123.              (dolist (handler *descriptor-handlers*)
  124.                (let ((fd (handler-descriptor handler)))
  125.                  (if (plusp (ecase (handler-direction handler)
  126.                               (:input (fd-isset fd rfd))
  127.                               (:output (fd-isset fd wfd))))
  128.                      (funcall (handler-function handler)
  129.                               (handler-descriptor handler))))))))))))
  130.  
  131.  
  132. ;;; Wait for up to timeout seconds for an event to happen. Make sure all
  133. ;;; pending events are processed before returning.
  134. (defun serve-all-events (&optional (timeout 0))
  135.   "SERVE-ALL-EVENTS calls SERVE-EVENT with the specified timeout. If
  136. SERVE-EVENT does something (returns T) it loops over SERVE-EVENT with a
  137. timeout of 0 until there are no more events to serve. SERVE-ALL-EVENTS returns
  138. T if SERVE-EVENT did something and NIL if not."
  139.   (do ((res nil)
  140.        (sval (serve-event timeout) (serve-event 0)))
  141.       ((null sval) res)
  142.     (setq res t)))
  143.  
  144.  
  145. ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
  146. ;; Test Example
  147. ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
  148.  
  149. ;; (defun test-stdin ()
  150. ;;   (format t "DOING STDIN~%")
  151. ;;   (with-fd-handler (0 :input #'(lambda (fd) (declare (ignore fd))
  152. ;;                                        (format t "Got data~%")
  153. ;;                                        (read-char)))
  154. ;;     (loop ;; FIXME: End condition
  155. ;;        (format t "Entering serve-all-events...~%")(force-output)
  156. ;;        (serve-all-events 5)
  157. ;;        (format t "Events served~%"))))

I've submitted a patch to the ECL list, and there's been some interest in integrating it so it should be part of it soon.

Update: This was added in the ECL 0.9j release.

Robust backups

Jamie Zawinski has just posted some instructions on a basic backup scheme. While the general advice is good, there are a couple of improvements that could be made, so I thought I'd describe my home backup system.

The main problem the the rsync script Jamie posted is that if you accidentally delete some important files the next time your backup runs rsync will delete the backups of those files. This is due to the --delete command. The obvious solution is to remove that option, but that will tend to leave a very messy filesystem over time. My preferred solution is to use the --backup and --backup-dir options to create a simple form of incremental backup. My script looks like this:

Bash Script:
  1. dir=/media/backup
  2. stamp=`date +'%Y%m%d'`
  3. bdir=$dir/.incremental.$stamp
  4.  
  5. rsync -Pav --backup --backup-dir=$bdir --delete-after \
  6.      /home /etc /usr/local /opt \
  7.      --exclude-from=/home/ssmith/.backup-exclude  \
  8.     $dir

Note that my script doesn't do a full-drive image like Jamie's as I don't find that important. Also, my script excludes some files I don't care about (i.e. can replace easily). Regardless, the principle is the same.

However this doesn't cover the house-burns-down scenario. Obviously the above script could easily be used with a drive you then bring to work as with JWZ's suggestion. But in my experience you will simply never do that.

The solution? I use a near-identical script to backup to a remote host. If you already have a machine somewhere you can use that, or there are many remote backup services, some of which have a direct rsync service. While the initial upload will be pretty large the subsequent incremental uploads will be much quicker, especially if you have ADSL2+, which has 1Mbit uploads. The cool part is that this leverages the uploads-are-free aspect of most Australian ISPs.

The only issue with this is that your data is possibly exposed to third-parties. My normal solution to this is to place any critical data in a directory that is encrypted using encfs. However I'm now having a look at Duplicity as an alternative, which works in much the same way but has GnuPG signing and encryption. It also supports more protocols, such as webdav and Amazon's S3 service.

  1. Archives

  2. Categories

  3. Twitter

    • Just got pictures of the earthquake damage from my sister in Kaiapoi. Real "the ground opened up" stuff. 16 minutes ago
    • Ooo, the beta of Angry Birds is out on Android. 17 hours ago
    • Released a new version of my Android Internode widget; fixes the problem with Internode's new SSL cert. 22 hours ago
    • @wangjammer5: Cool, here's something to get you started: http://is.gd/eT33b :) 1 day ago
    • So, if iTune Ping is Apple's social network will the perpetual Apple tweet circle-jerk move there now? 1 day ago
  4. RSS Google Reader Shared Items