Python encoding issue when trying to parse JSON tweets -

-

i trying parse out tweet , username sections of json object returned twitter using following code: 

class listener(streamlistener):    def on_data(self, data):           all_data = json.loads(data)           tweet = all_data["text"]           username = all_data["user"]["screen_name"]            c.execute("insert tweets (tweet_time, username, tweet) values (%s,%s,%s)" ,                     (time.time(), username, tweet))           print (username, tweet)           return true    def on_error(self, status):       print (status)  auth = oauthhandler(ckey, csecret) auth.set_access_token(atoken, asecret) twitterstream = stream(auth, listener()) twitterstream.filter(track = ["lebron james"])

but following error. how can code adjusted decode or encode response properly?

traceback (most recent call last):    file "c:/users/sagars/pycharmprojects/youtube nlp lessons/twitter stream db.py", line 45, in <module>     twitterstream.filter(track = ["lebron james"])   file "c:\python34\lib\site-packages\tweepy\streaming.py", line 428, in filter     self._start(async)   file "c:\python34\lib\site-packages\tweepy\streaming.py", line 346, in _start     self._run()   file "c:\python34\lib\site-packages\tweepy\streaming.py", line 286, in _run     raise exception   file "c:\python34\lib\site-packages\tweepy\streaming.py", line 255, in _run     self._read_loop(resp)   file "c:\python34\lib\site-packages\tweepy\streaming.py", line 309, in _read_loop     self._data(next_status_obj)   file "c:\python34\lib\site-packages\tweepy\streaming.py", line 289, in _data     if self.listener.on_data(data) false:   file "c:/users/sagars/pycharmprojects/youtube nlp lessons/twitter stream db.py", line 36, in on_data     print (username, tweet)   file "c:\python34\lib\encodings\cp1252.py", line 19, in encode     return codecs.charmap_encode(input,self.errors,encoding_table)[0] unicodeencodeerror: 'charmap' codec can't encode characters in position 0-8: character maps <undefined>

unfortunately problem information twitter not utf-8 encoded, causing charmap error. fix that, you'll need encode it.

tweet = all_data["text"].encode('utf-8') username = all_data["user"]["screen_name"].encode('utf-8')

this cause lose of emoji , special characters show in tweet, converted \x899. if need information (i discard myself) sentiment analysis, you'll need install package pre-compiled list convert them accordingly.


Comments

Post a Comment

Popular posts from this blog

PHP DOM loadHTML() method unusual warning -

-
i had following code works fine on localhost: $document = new domdocument(); $document->loadhtml($p_result); $form = $document->getelementsbytagname('form')->item(0); // code continues using $form variable after same code updated on outside server, loadhtml() failed , issued warning. warning: domdocument::loadhtml() expects parameter 1 valid path, string given in path/name/to/script.php returned null instead of object code pretty gets fatal error. note contents of $p_result absolutely same on outside server , on localhost. but why displays kind of warning , why not work? doesn't loadhtml() expects argument 1 string in first place? why method expects parameter 1 valid path ? just make clear i'm not calling loadhtmlfile() , i'm calling loadhtml() . thanks. you're affected one of php bugs . issue present in php 5.6.8 , 5.6.9. have affected php version on server, , bug-free version on localhost. the bug forbids null
Read more

python - How to create jsonb index using GIN on SQLAlchemy? -

-
here's current code creating index jsonb. index("mytable_data_idx_id_key", mytable.data['id'].astext, postgresql_using='gin') but got error. sqlalchemy.exc.programmingerror: (psycopg2.programmingerror) data type text has no default operator class access method "gin" hint: must specify operator class index or define default operator class data type. [sql: "create index event_data_idx_id_key on event using gin ((data ->> 'id'))"] is there way create index on sqlalchemy? the postgresql specific sqlalchemy docs @ http://docs.sqlalchemy.org/en/latest/dialects/postgresql.html#operator-classes mention postgresql_ops dictionary provide "operator class" used postgresql, , provide example illustrating use: index('my_index', my_table.c.id, my_table.c.data, postgresql_ops={ 'data': 'text_pattern_ops',
Read more

c# - TransactionScope not rolling back although no complete() is called -

-
i'm using transactionscope rollback transaction fail bool errorreported = false; action<importerrorlog> newerrorcallback = e => { errorreported = true; errorcallback(e); }; using (var transaction = new transactionscope()) { foreach (importtaskdefinition task in taskdefinition) { loader.load(streamfile, newerrorcallback, task.destinationtable, processingtaskid); } if (!errorreported) transaction.complete(); } i'm sure there no transactionscope started ahead or after code. i'm using entity framework insert in db. regardless state of errorreported transaction never rolled in case of error. what missing ? transactionscope sets transaction.current . that's does. wants transacted must @ property. i believe ef each time connection opened reason. probably, connection open when scope installed. open connection inside of scope or enlist manually. ef has nasty "design decision": default opens
Read more