You Are Here:

Community: Developer Discussion Boards

#1 Old Question loss of precision of floats values when stored in a e32db - 2006-02-22, 11:01

Join Date: Jan 2006
Posts: 3
madmartygan
Offline
Registered User
Dear All,
I hope you can help me to understand what is going on on my code.
The problem I am facing is that I have an x,y value that are floats with many digits after the comma. I am storing these values into an e32db.
When I am retrieving these values from the db, it seems many of the digits after the comma have been truncated. For example:
A number like 9.15435791015625 becomes 9.154358.
The problem is that I want to keep all the digits because, dropping them will reflect into a serious problem of precision for my application.

Some other information on my implementation.
1) I am defining the values as 'DOUBLE PRECISION' on the db definition, sql CREATE;
2) I am using python floats into the sql insert:
Code:
INSERT INTO Message (ID, uid, username, x, y, time, tipe, subject, content, parentID) VALUES (%d,%d,'%s',%f,%f,#%s#,'%s','%s','%s',%d)"%(
            self.last_id,
            uid,
            self.username,
            self.x,
            self.y,
            ....
3) the sql select is performed using the following method:
Code:
     def select_all(self, query):
        self.dbv.prepare(self.db, unicode(query))
        self.dbv.first_line()
        result = []
        col_count = self.dbv.col_count()
        for i in range(self.dbv.count_line()):
            self.dbv.get_line()
            row = [self.dbv.col(i+1) for i in range(col_count)]
            result.append(row)
            self.dbv.next_line()
        return result
To solve this I was seriously considering to use VARCHAR instead of FLOAT or DOUBLE PRECISION. Has anyone faced the same problem and possibly found a solution?
Thanks a trillion


Mauro

http://www.i-cherubini.it/mauro/blog/
Reply With Quote
Reply « Previous Thread | Next Thread »
Display Modes
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Rate This Thread
Rate This Thread:

Posting Rules

You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is Off
HTML code is Off
Forum Jump
Similar Threads
Thread Thread Starter Forum Replies Last Post
Error: possible loss of precision... aldrin002 Mobile Java Media (Graphics & Sounds) 1 2003-08-26 09:01

Rate This

 
Bookmark this page: DeliciousDiggFacebookGoogleYahooStumbleUponRedditDiigoTechnocratiTwitter  Share this page Share this page Print this Page Print this page Invite a friend Invite a friend
京ICP备05048969号    Email Newsletters Press Terms & Conditions Privacy Policy Sitemap Contact Us © 2009 Nokia