Relocate wiki from trac

New module file_io which takes care of this.  All output should go
through file_io.write, which encodes it using the appropriate
character encoding.  For output to a temporary file, use "with
file_io.tmpfile('name')".

Moved WorblehatException to new module exc, so all modules can access
it without importing worblehat.

README.md

@@ -0,0 +1,35 @@
+![](./wiki/graphics/project_icon.png)
+
+# Worblehat
+
+Worblehat er PVVs biblioteksystem.
+
+- [FAQ](./wiki/faq.md)
+
+## Foreløpige planer
+
+### Database
+
+Vi skal ha en PostgreSQL-database, med designet gitt i [`db.txt`](./db.txt). Er designet feil, vennligst rett.
+
+### Informasjonsinnhenting
+
+Tenkte å bruke protokollen Z39.50 mot BIBSYS. Vi skal ha et kommandolinjegrensesnitt for å legge inn bøker. Vi skal kjøpe en strekkodeleser for å blippe inn ISBN-nummeret, og systemet skal slå opp dette i BIBSYS. Resultatene skal returneres, og vi skal velge riktig bok, som så skal bli lagt inn i databasen. Dette bør det være mulig å gjøre både brukervennlig for registrering av en bok, og raskt ved registrering av flere hundre bøker.
+
+### Utspørring
+
+Kommandolinjegrensesnitt, IRC-bot og/eller web-grensesnitt for å spørre databasen. IRC-boten skal ha et logikklag mellom seg selv og databasen, for å slippe så mye logikk internt. Dette laget kan også de andre grensesnittene bruke.
+
+### Ressurser
+
+#### Informasjon om bøkene
+
+**Z39.50**
+
+- https://www.bibsys.no/wps/wcm/connect/BIBSYS+Eng/Main+Menu/Search/Z39.50+Service+Description
+- https://www.norzig.no/
+
+**Klassifikasjon**
+
+- https://en.wikipedia.org/wiki/Universal_Decimal_Classification
+- https://en.wikipedia.org/wiki/Dewey_Decimal_Classification

cli/exc.py

@@ -0,0 +1,3 @@
+class WorblehatException(Exception):
+    def __init__(self, msg):
+        Exception.__init__(self, msg)

cli/file_io.py

@@ -0,0 +1,79 @@
+import os
+import tempfile
+import locale
+import sys
+from exc import WorblehatException
+
+stdout_encoding = locale.getpreferredencoding()
+file_encoding = 'utf-8'
+
+output = sys.stdout
+
+# def write_tmpfile(pfix, content, encoding='utf8'):
+#     file = tempfile.NamedTemporaryFile(prefix=pfix+'-', dir='/tmp', delete=False)
+#     file.write(content.encode(encoding))
+#     name = file.name
+#     file.close()
+#     return name
+
+def open_tmpfile(prefix):
+    global output
+    if output != sys.stdout:
+        raise WorblehatException('open_tmpfile: Already writing to a file')
+    tmpfile = tempfile.NamedTemporaryFile(prefix='worblehat-%s-' % prefix,
+                                          dir='/tmp',
+                                          delete=False)
+    output = tmpfile
+    return tmpfile.name
+
+def close_tmpfile():
+    global output
+    if output == sys.stdout:
+        raise WorblehatException('close_tmpfile: No file open')
+    output.close()
+    output = sys.stdout
+    
+def output_encoding():
+    if output == sys.stdout:
+        return stdout_encoding
+    else:
+        return file_encoding
+
+def encoding_comment():
+    return '# -*- coding: %s -*-\n' % file_encoding
+
+def write(data):
+    if type(data) == unicode:
+        data = data.encode(output_encoding())
+    output.write(data)
+
+def write_stderr(data):
+    if type(data) == unicode:
+        data = data.encode(output_encoding())
+    sys.stderr.write(data)
+
+debugging = True
+def debug(msg):
+    if debugging:
+        write_stderr('DEBUG: %s\n' % msg)
+
+def tmpfile_name():
+    if output == sys.stdout:
+        raise WorblehatException('tmpfile_name: No file open')
+    return output.name
+
+class tmpfile:
+    def __init__(self, prefix):
+        self.prefix = prefix
+    def __enter__(self):
+        open_tmpfile(self.prefix)
+    def __exit__(self, exc_type, exc, traceback):
+        close_tmpfile()
+
+def run_editor(filename):
+    if os.path.exists(filename):
+        os.system("%s %s || /usr/bin/env vi %s" %
+                  (os.getenv("EDITOR"), filename, filename))
+    else:
+        exit("Error: %s: File does not exist!" % filename)
+

cli/fileformat.py

@@ -1,5 +1,6 @@
 import re
 import types
+from exc import WorblehatException
 
 # The possible fields for each type of object.
 #
@@ -52,10 +53,13 @@ action_fields = {
         { 'type': 'category',
           'required': ['id'] } }
 
-class CommitFormatSyntaxError(Exception):
+class CommitFormatSyntaxError(WorblehatException):
     def __init__(self, msg, linenr):
-        super(CommitFormatSyntaxError, self).__init__(self, 'Syntax error on line %d: %s' %
+        WorblehatException.__init__(self, 'Syntax error on line %d: %s' % (linenr, msg))
-                                                      (linenr, msg))
+
+class CommitFormatError(WorblehatException):
+    def __init__(self, msg):
+        WorblehatException.__init__(self, msg)
 
 def read_field_value_str(val):
     if val.strip() == '':
@@ -65,7 +69,10 @@ def read_field_value_str(val):
 def read_field_value_int(val):
     if val.strip() == '':
         return None
-    return int(val.strip())
+    try:
+        return int(val.strip())
+    except ValueError, TypeError:
+        raise WorblehatException('%s is not an integer' % val)
 
 def read_field_value_dict(val):
     d = {}
@@ -171,23 +178,27 @@ def write_field_value_str(val):
     lines = ''
     if not val:
         val = ''
-    val = unicode(val)
     value_lines = val.split('\n')
     for l in value_lines:
         lines += ' ' + l + '\n'
+    if len(value_lines) > 1:
+        lines = '\n' + lines
     return lines
 
+def write_field_value_int(val):
+    return ' %s\n' % str(val)
+
 def write_field_value_dict(val):
     lines = '\n'
     for (key,values) in val.items():
         for single_value in values:
-            lines += ' ' + key + ' ' + unicode(single_value) + '\n'
+            lines += ' ' + key + ' ' + single_value + '\n'
     return lines
 
 def write_field_value_list(val):
     lines = ''
     for single_value in val:
-        lines += ' ' + unicode(single_value)
+        lines += ' ' + single_value
     return lines
 
 def make_comment(s):
@@ -197,7 +208,7 @@ def make_comment(s):
 def write_action(d):
     if type(d) in types.StringTypes:
         return make_comment(d)
-    lines = ''
+    lines = u''
     if 'comment' in d:
         lines += make_comment(d['comment'])
     action = d['action']
@@ -206,7 +217,7 @@ def write_action(d):
     for field, ftype in fields[data_type]:
         if field in d:
             value_writer = {'s': write_field_value_str,
-                            'i': write_field_value_str,
+                            'i': write_field_value_int,
                             'd': write_field_value_dict,
                             'l': write_field_value_list}[ftype]
             lines += field + ':' + value_writer(d[field])

cli/util.py

@@ -1,12 +1,32 @@
-import os
+db_encoding = 'utf8'
-import tempfile
+
+def value_to_db(value):
+    if type(value) == unicode:
+        return value.encode(db_encoding)
+    return value
+
+def value_from_db(value):
+    if type(value) == str:
+        return unicode(value, db_encoding)
+    return value
 
 def execute_query(cursor, query, bindings):
     for (key, val) in bindings.items():
         if val == None:
             query = query.replace('%(' + key + ')d', 'NULL')
+    bindings = map_dict(value_to_db, bindings)
     cursor.execute(query, bindings)
 
+def fetchone(cursor):
+    a = cursor.fetchone()
+    if a != None:
+        return map(value_from_db, a)
+    return None
+
+def fetchall(cursor):
+    return map(lambda row: map(value_from_db, row),
+               cursor.fetchall())
+
 def make_result_dict(cursor, row):
     d = {}
     for i in xrange(len(row)):
@@ -14,14 +34,14 @@ def make_result_dict(cursor, row):
     return d
 
 def fetchone_dict(cursor):
-    row = cursor.fetchone()
+    row = fetchone(cursor)
     if row != None:
         return make_result_dict(cursor, row)
     return None
 
 def fetchall_dict(cursor):
     return map(lambda r: make_result_dict(cursor, r),
-               cursor.fetchall())
+               fetchall(cursor))
 
 def first(lst):
     return lst[0]
@@ -56,6 +76,12 @@ def mapcond(fun, predicate, lst):
         return x
     return map(mapfun, lst)
 
+def map_dict(fun, d):
+    res = {}
+    for key in d:
+        res[key] = fun(d[key])
+    return res
+
 def maptup(fun, lst):
     return tuple(map(fun, lst))
 
@@ -84,16 +110,3 @@ def combine_dicts(*dicts):
         res.update(d)
     return res
 
-def run_editor(filename):
-    if os.path.exists(filename):
-        os.system("%s %s || /usr/bin/env vi %s" %
-                  (os.getenv("EDITOR"), filename, filename))
-    else:
-        exit("Error: %s: File does not exist!" % filename)
-
-def write_tmpfile(pfix, content, encoding='utf8'):
-    file = tempfile.NamedTemporaryFile(prefix=pfix+'-', dir='/tmp', delete=False)
-    file.write(content.encode(encoding))
-    name = file.name
-    file.close()
-    return name

cli/worblehat.py

@@ -7,6 +7,8 @@ import pgdb
 from fileformat import read_actionlist, write_actionlist
 from google_interface import google_suggest_book_data
 from util import *
+from exc import WorblehatException
+from file_io import tmpfile, tmpfile_name, write, write_stderr, debug, run_editor, encoding_comment
 
 # connection = pgdb.connect(database='oysteini_pbb2',
 #                           user='oysteini_pbb',
@@ -57,6 +59,8 @@ q_edit_person = \
     'UPDATE person ' \
     'SET lastname=%(lastname)s, firstname=%(firstname)s ' \
     'WHERE id=%(id)s'
+q_delete_person = \
+    'DELETE FROM person WHERE id=%(id)s'
 q_new_book = \
     'INSERT INTO book ' \
     '            (isbn, id, title, subtitle, category, publisher, ' \
@@ -85,7 +89,7 @@ q_edit_category = \
     'WHERE id=%(id)s'
 q_add_bookreference = \
     'INSERT INTO bookreference (book, reftype, value) ' \
-    'VALUES (%(isbn)s, %(reftype)s, %(value))'
+    'VALUES (%(isbn)s, %(reftype)s, %(value)s)'
 
 def connect_to_db():
     connection = pgdb.connect(database='oysteini_pbb2',
@@ -102,7 +106,7 @@ def get_by_id(connection, id):
     for (typ,q) in [('book', q_book),
                     ('person', q_person),
                     ('category', q_cat)]:
-        c.execute(q, {'id': id})
+        execute_query(c, q, {'id': id})
         if c.rowcount > 0:
             d = fetchone_dict(c)
             d['type'] = typ
@@ -214,15 +218,16 @@ def show(connection, ids, commit_format=False, tmp_file=False):
             objects[i] = show_fun(objects[i])
 
     if commit_format:
-        output = write_actionlist(objects)
+        output = encoding_comment() + write_actionlist(objects)
     else:
         output = '\n'.join(objects)
     if tmp_file:
-        filename = write_tmpfile('.'.join(ids), output)
+        with tmpfile('.'.join(ids)):
-        print filename
+            write_stderr('%s\n' % tmpfile_name())
-        return filename
+            write(output)
+            return tmpfile_name()
     else:
-        print output.strip()
+        write(output)
 
 def list_books(connection):
     c = connection.cursor()
@@ -230,7 +235,7 @@ def list_books(connection):
     #print fetchall_dict(c)
     for i in xrange(c.rowcount):
         book = fetchone_dict(c)
-        print('%-13s %-10s %-60s %s' %
+        write('%-13s %-10s %-60s %s\n' %
               (book['isbn'], str_or_empty(book['id']),
                cut_str(book['title'], 60), book['persons']))
 
@@ -239,16 +244,16 @@ def list_persons(connection):
     c.execute(q_list_persons)
     for i in xrange(c.rowcount):
         person = fetchone_dict(c)
-        print '%-5s %-30s %d books' % (person['id'],
+        write('%-5s %-30s %d books\n' % (person['id'],
                                        person['firstname']+' '+person['lastname'],
-                                       person['num_books'])
+                                       person['num_books']))
 
 def list_categories(connection):
     c = connection.cursor()
     c.execute(q_list_categories)
     for i in xrange(c.rowcount):
         cat = fetchone_dict(c)
-        print '%-15s %-30s %d books' % (cat['id'], cat['name'], cat['num_books'])
+        write('%-15s %-30s %d books\n' % (cat['id'], cat['name'], cat['num_books']))
 
 def list_cmd(connection, what):
     funs = { 'book': list_books,
@@ -260,9 +265,11 @@ def list_cmd(connection, what):
 def search_book(connection, search_strings, search_description=False):
     c = connection.cursor()
     if search_description:
-        where_clauses = ['book.title ILIKE %s OR book.subtitle ILIKE %s OR book.series ILIKE %s OR person.lastname ILIKE %s OR person.firstname ILIKE %s OR book.description ILIKE %s']*len(search_strings)
+        where_clauses = ['book.title ILIKE %s OR book.subtitle ILIKE %s OR book.series ILIKE %s \
+                            OR person.lastname ILIKE %s OR person.firstname ILIKE %s OR book.description ILIKE %s']*len(search_strings)
     else:
-        where_clauses = ['book.title ILIKE %s OR book.subtitle ILIKE %s OR book.series ILIKE %s OR person.lastname ILIKE %s OR person.firstname ILIKE %s']*len(search_strings)
+        where_clauses = ['book.title ILIKE %s OR book.subtitle ILIKE %s OR book.series ILIKE %s \
+                            OR person.lastname ILIKE %s OR person.firstname ILIKE %s']*len(search_strings)
 
     result_list = []
     for s in search_strings:
@@ -272,10 +279,18 @@ def search_book(connection, search_strings, search_description=False):
         else:
             for i in range(5):
                 result_list.append(s)
-    c.execute('SELECT * FROM book LEFT JOIN bookperson ON book.isbn=bookperson.book LEFT JOIN person ON person.id=bookperson.person WHERE ' + ' OR '.join(where_clauses), map(lambda s:'%' + s + '%',result_list)) 
+    c.execute('SELECT isbn,book.id AS id,title,category, \
+                array_to_string(array_agg(person.lastname || \' (\' || person.id || \')\'), \', \') AS persons \
+                FROM book LEFT JOIN bookperson ON book.isbn=bookperson.book \
+                LEFT JOIN person ON person.id=bookperson.person \
+                WHERE ' + ' OR '.join(where_clauses) + '\
+                GROUP BY isbn, book.id, title, category \
+                ', map(lambda s:'%' + s + '%',result_list)) 
     for  i in xrange(c.rowcount):
         book = fetchone_dict(c)
-        print book['isbn'], book['title'], book['person']
+        write('%-13s %-10s %-60s %s' %
+              (book['isbn'], str_or_empty(book['id']),
+               cut_str(book['title'], 60), book['persons']))
 
 
 def search_person(connection, search_strings):
@@ -284,13 +299,14 @@ def search_person(connection, search_strings):
     for s in search_strings:
         for i in range(3):
             result_strings.append(s)
-    c.execute('SELECT * FROM person LEFT JOIN bookperson ON person.id=bookperson.person WHERE person.lastname ILIKE %s or person.firstname ILIKE %s OR person.id ILIKE %s', result_strings)
+    c.execute('SELECT * FROM person LEFT JOIN bookperson ON person.id=bookperson.person \
+                WHERE person.lastname ILIKE %s or person.firstname ILIKE %s OR person.id ILIKE %s', result_strings)
     for i in xrange(c.rowcount):
         person = fetchone_dict(c)
-        print person['lastname'], ', ', person['firstname'], '\t', person['book']
+        write(person['lastname'], ', ', person['firstname'], '\t', person['book'])
 
 def do_action(connection, action):
-    print 'ACTION %s ' % action
+    debug('ACTION %s ' % action)
     c = connection.cursor()
     queries = {'new-person': q_new_person,
                'edit-person': q_edit_person,
@@ -301,9 +317,9 @@ def do_action(connection, action):
     action_type = action['action']
     execute_query(c, queries[action_type], action)
     if action_type in ['new-book', 'edit-book']:
-        print 'FIXING PERSONS: REMOVING'
+        debug('FIXING PERSONS: REMOVING')
         c.execute(q_remove_bookpersons, {'isbn': action['isbn']})
-        print 'FIXING PERSONS: ADDING'
+        debug('FIXING PERSONS: ADDING')
         if action['persons']:
             for (relation, personlist) in action['persons'].items():
                 for person in personlist:
@@ -313,30 +329,23 @@ def do_action(connection, action):
                                'relation': relation})
         if action['references']:
             c.execute('SELECT referencetype.id FROM referencetype')
-            refs = c.fetchone()
+            legal_reftypes = [a for list in c.fetchall() for a in list]
             for (reftype, reflist) in action['references'].items():
                 for ref in reflist:
-                    if ref in refs:
+                    if reftype in legal_reftypes:
-                        c.execute(q_add_reference,
+                        c.execute(q_add_bookreference,
                                   {'isbn': action['isbn'],
                                    'reftype': reftype,
                                    'value': ref})
                     else:
                         raise WorblehatException('%s is not in the defined references, please use a more general one' % reftype)
 
-class WorblehatException(Exception):
-    def __init__(self, msg):
-        Exception.__init__(self, msg)
-
-
 def commit_actions(connection, actions):
     for action in actions:
         try:
             do_action(connection, action)
         except pgdb.DatabaseError, err:
-            print>>sys.stderr, 'Error in "%s" action: %s' % (action['action'], err)
+            raise WorblehatException('commit: Error in "%s" action: %s' % (action['action'], err))
-            print>>sys.stderr, 'Giving up.'
-            return
     connection.commit()
 
 def commit(connection, filename=None):
@@ -346,9 +355,7 @@ def commit(connection, filename=None):
             text = f.read()
             f.close()
         except IOError, e:
-            print 'commit: Error reading file %s: %s' % (filename, e)
+            raise WorblehatException('commit: Error reading file %s: %s' % (filename, e))
-            print 'Exiting.'
-            sys.exit(1)
     else:
         text = sys.stdin.read()
 
@@ -357,9 +364,17 @@ def commit(connection, filename=None):
 
 def edit(connection, ids):
     filename = show(connection, ids, commit_format=True, tmp_file=True)
-    print filename
+    done = False
-    run_editor(filename)
+    while not done:
-    commit(connection, filename)
+        run_editor(filename)
+        try:
+            commit(connection, filename)
+            done = True
+        except WorblehatException, exc:
+            write_stderr('%s\n' % exc)
+            answer = raw_input('Retry? [Y/n] ')
+            if answer == 'n':
+                done = True
 
 def map_cmd(connection, shelfname=None, category=None):
     pass
@@ -367,17 +382,15 @@ def map_cmd(connection, shelfname=None, category=None):
 def suggest_book_data(connection, tmp_file=False):
     return google_suggest_book_data(connection, tmp_file)
 
-<<<<<<< .mine
 def register_books(connection):
     filename = suggest_book_data(connection, tmp_file=True)
+    #print("Tempfile filename: " + filename)
     run_editor(filename)
     commit(connection, filename)
 
-=======
 def give_bananas():
-    print "Om nom nom... Thanks!"
+    write("Om nom nom... Thanks!")
 
->>>>>>> .r164
 commands = { 'show':
                  { 'args': [('ids', (1,None))],
                    'options': ['commit_format', 'tmp_file'],

wiki/assets/møte_270209-referat

@@ -0,0 +1,53 @@
+Hei,
+
+i går startet vi opp med planleggingen av det nye biblioteksystemet. Det
+har fått navnet Worblehat, og mailinglista worblehat@pvv.ntnu.no er
+opprettet. Siden det fremdeles ikke er så mange som har meldt seg på
+der, spammer jeg denne også til aktive.
+
+Det er dessuten sendt mail til drift, om å legge oss ut på dev.
+
+Vi har laget et foreløpig design av både systemet og databasen. Disse er
+foreløpig å finne på http://www.pvv.ntnu.no/~tirilane/worblehat.
+Rettelser og forbedringer mottas med takk. Det hadde vært spesielt fint
+om noen med peiling på databaser kunne gå gjennom worblehat-db.dia og
+fikse. Og det hadde vært enda bedre om noen med peiling på databaser
+kunne implementere databasen, og legge inn alle de magiske triksene for
+å få til til å gå raskt.
+
+Ellers er designet ganske enkelt. Vi trenger en database, og noen
+klienter. Vi trenger blant annet en for registrering av nye bøker. Den
+skal ha to modus:
+1) Registrering av en bok
+2) Registrering av mange bøker
+Vi tenkte at 1) kunne være sånn fin og brukervennlig, mens 2) skal gå
+raskt for mange bøker. Gjerne slik:
+Blippe bok, klienten plukker opp ISBN-nummeret, spør ISBNdb.com, og
+registrerer boka i databasen om den ble funnet. Ble den ikke funnet, må
+den på en eller annen måte si fra, slik at vi kan ta boka manuelt. Dette
+kan løses ved å samle opp ISBN-nummeret, men siden det å manuelt lete
+etter bøker med et gitt ISBN-nummer ikke akkurat er lett, hadde det nok
+vært bedre om den stoppet opp eller noe.
+
+Dessuten trenger vi noe for å spørre databasen. Her tenkte vi CHI,
+IRC-bot og noe web-basert. Strengt tatt trenger vi bare en, og da bør
+kanskje CLI ha hovedprioritet.
+
+For at en eventuell IRC-bot ikke skal gjøre for mye arbeid, tenkte vi å
+ha et lag mellom boten og databasen. Denne skal implementere
+søke-funksjonene, som boten bare trenger å kalle. De andre (CLI og web)
+står også fritt til å benytte seg av dette. Det blir opp til de som
+implementerer. Fordelen er mindre duplisering av kode. Ulempen er at da
+må alle skrives i samme språk. Men opp til implementørene.
+
+En ting vi ikke har bestemt, er om plasseringen i hylla skal baseres på
+Deweys system eller UDC. Fordelen med Dewey er at nummeret kan hentes
+ned fra ISBNdb.com sammen med resten av informasjonene. Fordelen med UDC
+er at det er mye mer spesialisert, hvilket kan være en stor fordel når
+vi har så mange bøker om et såpass smalt felt. Dessuten bruker Teknisk
+hovedbibliotek UDC. Hva synes man?
+http://en.wikipedia.org/wiki/Universal_Decimal_Classification
+http://en.wikipedia.org/wiki/Dewey_Decimal_Classification
+
+Og hvem har lyst til å fikse hva? Noen frivillige?
+

wiki/faq.md

@@ -0,0 +1,4 @@
+# Worblehat FAQ
+
+Q: Hvor kommer navnet fra?
+A: Discworld, bibliotekarens tidligere navn. Se https://wiki.lspace.org/Horace_Worblehat