Zap2xml for ATSC in OpenELEC

  Thread Rating:
  • 2 Vote(s) - 4 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Post Reply
edit4ever Offline
Senior Member
Posts: 272
Joined: Mar 2016
Reputation: 15
Post: #421
OK - So with the changes coming in Kodi 18 and the issues with different OSs - I think I'm going to go back to the beginning and write this as a true kodi addon. It was thrown together from a simple python command line program that was actually derived from a perl version. Since kodi and tvheadend were and afterthought and I started this when I knew absolutely nothing about python (and I'm still technically a novice!) I think it wil lbe better to rewrite the whole thing (or big parts of it).

Hopefully that can bring a unfied working addon for all platforms. Since I have a two week trip coming up - I can hopefully get a lot done on the plane rides. :-)


Current Kodi addon projects: zap2xml, sd4tvh and tvh2kodi (tvh config from inside Kodi)
Testing ATSC single and dual tuners on RPi3, KII Pro and M8S
find quote
clarkss12 Offline
Fan
Posts: 337
Joined: Mar 2015
Reputation: 2
Post: #422
That will be awesome, thanks.
find quote
edit4ever Offline
Senior Member
Posts: 272
Joined: Mar 2016
Reputation: 15
Post: #423
One more thing... @stephr1 - I did test the 0.7.4-test version that I posted and it worked on my end - so with whatever chenages you need for the ubuntu setup it should pull down the xtra details info for you.

If you have a working version right now for basic data - you can manually edit the zap2xml.py file and make changes to lines 341-351 making them match lines 341-354 on the 0.7.4-test version.

It adds a try: around the xtra details downloads and hopefully moves on if it fails.


Current Kodi addon projects: zap2xml, sd4tvh and tvh2kodi (tvh config from inside Kodi)
Testing ATSC single and dual tuners on RPi3, KII Pro and M8S
find quote
stephr1 Offline
Junior Member
Posts: 48
Joined: Aug 2016
Reputation: 0
Post: #424
I'm thinking with some of the things that have gone on recently with the system, something related to the grabber might be "off", rather than the grabber itself. If I get some time in the next couple of days, I'll "flatten" the grabber (uninstall/delete any extraneous files) and then reinstall, mod the paths and settings as needed, and try again.

I'll let you know the results.

As usual, thx and cheers....

(2017-09-09 16:55)edit4ever Wrote:  One more thing... @stephr1 - I did test the 0.7.4-test version that I posted and it worked on my end - so with whatever chenages you need for the ubuntu setup it should pull down the xtra details info for you.

If you have a working version right now for basic data - you can manually edit the zap2xml.py file and make changes to lines 341-351 making them match lines 341-354 on the 0.7.4-test version.

It adds a try: around the xtra details downloads and hopefully moves on if it fails.
find quote
stephr1 Offline
Junior Member
Posts: 48
Joined: Aug 2016
Reputation: 0
Post: #425
UPDATE (9/15/17):

Update happened as normal last night/this morning with this msg near the end of the log file (in .kodi/addons/.....):

2017-09-15 03:31:12,978 Form:
<zbSearchForm GET http://tvschedule.zap2it.com/tvlistings/ZCSearch.do application/x-www-form-urlencoded
<TextControl(searchTerm=)>
<HiddenControl(searchType=simple) (readonly)>
<HiddenControl(aid=tvschedule) (readonly)>
<SubmitControl(=Search) (readonly)>>:Function: loginZAP :Line: 570
2017-09-15 03:31:12,982 Form:
<zbSearchFormAdv GET http://tvschedule.zap2it.com/tvlistings/ZCSearch.do application/x-www-form-urlencoded
<TextControl(searchTerm=)>
<SelectControl(searchField=[*name, episodeName, episodeName, description, crew])>
<SelectControl(searchGenre=[*, movie, sports, children, special, news])>
<CheckboxControl(searchHD=[true])>
<HiddenControl(searchType=advanced) (readonly)>
<HiddenControl(aid=tvschedule) (readonly)>
<SubmitControl(=Search) (readonly)>>:Function: loginZAP :Line: 570
2017-09-15 03:31:12,985 Form:
<zcLoginForm POST http://tvschedule.zap2it.com/tvlistings/...?category= application/x-www-form-urlencoded
<TextControl(username=)>
<PasswordControl(password=)>
<SubmitControl(<None>=Login) (readonly)>
<IgnoreControl(loginReset=<None>)>
<HiddenControl(zc-login-forwardURL=) (readonly)>>:Function: loginZAP :Line: 570
2017-09-15 03:31:14,765 Matched .*Logout of your.*:Function: loginZAP :Line: 591

...and this msg in .kodi/userdata/add_ons/.... at the end of the log file:

http://tvschedule.zap2it.com/tvlistings/...tvschedule

http://tvschedule.zap2it.com/tvlistings/...tvschedule

Matched .*Logout of your.*:Function: loginZAP :Line: 591
Getting: http://tvschedule.zap2it.com/tvlistings/...tvschedule
[55/56] Parsing: /home/osmc/.kodi/addons/script.module.zap2xml/cache/1506625200000.html.gz
Getting: http://tvschedule.zap2it.com/tvlistings/...tvschedule
[56/56] Parsing: /home/osmc/.kodi/addons/script.module.zap2xml/cache/1506646800000.html.gz
Downloaded 896852 bytes in 2 http requests.

Writing XML file: /home/osmc/.xmltv/tvguide.xml

Completed in 158s (Parse: 79s) 78 stations, 10729 programs, 36152 scheduled.

So, looks like success. I want to watch it for another few days and see how it works (or doesn't work). Looks like (for me, anyway) having xtra details enabled is causing problems. I'm wondering if something else (I have a Dish DVR that does nightly updates) is trying to use my local net around the same time and that's causing problems for the RPI 3/OSMC system. It shouldn't, but variables do and constants aren't Sad To confirm or deny this I'll check out "call home" times for both units and make sure they access at different times at night (This wasn't a problem until recently, tho Sad

Peace out and cheers....


ORIGINAL posting:

Not sure what's going on. I completely uninstalled the 0.7.4-test. Deleted the 2 script.module.zap2zml folders (left the tv_grab_zap2xml file since that did not seem to be an issue) and did a reinstall of V0.7.4-test. I have had mixed results. I was able to do a couple of manual updates (restart tvheadend) over the past few days with no errors. However, the normal 3:30 AM updates are not happening and are producing errors. And a recent manual update failed.

In the .kodi/addons... folder I'm still getting the "HTTP Error 500: Internal Server Error" msgs. In the log file under .kodi/userdate.... I'm getting the following msgs (Can't tell if it's directly related to the Error 500 msg. Based on the 3 retries, I expect it is.):

Getting: http://tvschedule.zap2it.com/tvlistings/...3657560349
:Function: getURL :Line: 225
error<class 'mechanize._response.httperror_seek_wrapper'>
Getting: http://tvschedule.zap2it.com/tvlistings/...3657560349
:Function: getURL :Line: 225
error<class 'mechanize._response.httperror_seek_wrapper'>
Getting: http://tvschedule.zap2it.com/tvlistings/...3657560349
:Function: getURL :Line: 225
error<class 'mechanize._response.httperror_seek_wrapper'>
Failed to download within 3 retries.

this was done with xtra details enabled. So, I've disabled those again and will try both a manual run as well as see what happens overnight.

I'll post back results tomorrow.

Thanks and cheers....

(2017-09-10 17:58)stephr1 Wrote:  I'm thinking with some of the things that have gone on recently with the system, something related to the grabber might be "off", rather than the grabber itself. If I get some time in the next couple of days, I'll "flatten" the grabber (uninstall/delete any extraneous files) and then reinstall, mod the paths and settings as needed, and try again.

I'll let you know the results.

As usual, thx and cheers....

(2017-09-09 16:55)edit4ever Wrote:  One more thing... @stephr1 - I did test the 0.7.4-test version that I posted and it worked on my end - so with whatever chenages you need for the ubuntu setup it should pull down the xtra details info for you.

If you have a working version right now for basic data - you can manually edit the zap2xml.py file and make changes to lines 341-351 making them match lines 341-354 on the 0.7.4-test version.

It adds a try: around the xtra details downloads and hopefully moves on if it fails.
(This post was last modified: 2017-09-15 19:26 by stephr1.)
find quote
edit4ever Offline
Senior Member
Posts: 272
Joined: Mar 2016
Reputation: 15
Post: #426
I finally am back home and get back to testing and reworking this issue.

@stephr1 - since I can not recreate the issue on my setup (my guide data loads correctly, including xtra details)...I would like to test if I can recreate using your lineup. Can you let me know what zipcode and lineup you are using that has the error? I will test and see if the same thing happens on my end if I use your lineup.

Thanks!


Current Kodi addon projects: zap2xml, sd4tvh and tvh2kodi (tvh config from inside Kodi)
Testing ATSC single and dual tuners on RPi3, KII Pro and M8S
find quote
stephr1 Offline
Junior Member
Posts: 48
Joined: Aug 2016
Reputation: 0
Post: #427
I'm at 94086.

I'm having mixed results. Since my last posting, in order I had 1 successful updt, 2 unsuccessful, followed by unsuccessful last night.

I'm kinda wondering back to the failed updt when this all began. I have no idea to be sure, but I'm still sensing some kind of network conflict or other related issue. I've come close a couple of times to flattening the system and doing a new build (after salvaging the movies and tv shows already on the system). Not quite yet (cuz I'm optimistic Wink, but......

Thanks for your continued persistence on this. Maybe we'll get it figured out.

(2017-09-21 01:00)edit4ever Wrote:  I finally am back home and get back to testing and reworking this issue.

@stephr1 - since I can not recreate the issue on my setup (my guide data loads correctly, including xtra details)...I would like to test if I can recreate using your lineup. Can you let me know what zipcode and lineup you are using that has the error? I will test and see if the same thing happens on my end if I use your lineup.

Thanks!
find quote
edit4ever Offline
Senior Member
Posts: 272
Joined: Mar 2016
Reputation: 15
Post: #428
94086 and an OTA (antenna) lineup?


Current Kodi addon projects: zap2xml, sd4tvh and tvh2kodi (tvh config from inside Kodi)
Testing ATSC single and dual tuners on RPi3, KII Pro and M8S
find quote
edit4ever Offline
Senior Member
Posts: 272
Joined: Mar 2016
Reputation: 15
Post: #429
I just ran a full day download with xtra details for all channels in the 94086 OTA lineup. Built the xmltv.xml file just fine.

I'm guessing the error is caused when details data is missing. I'll let this continue for a few days to see if the error shows up. Please let me know if you see an issue on your end.

Thanks!


Current Kodi addon projects: zap2xml, sd4tvh and tvh2kodi (tvh config from inside Kodi)
Testing ATSC single and dual tuners on RPi3, KII Pro and M8S
find quote
stephr1 Offline
Junior Member
Posts: 48
Joined: Aug 2016
Reputation: 0
Post: #430
Sorry. Yes, OTA.

Mine updated last night, as well. However, I have extra details disabled.

My plan was to send you the associated code (EP0XXX...XXX) for the data (or lack there of) that shows up when errors are flagged in the log file(s). I'll have to wait until I get an error (or see previous posting(s)). Maybe I should re-enable xtra details to speed up the process. I'll check the results tomorrow morning and decide at that point.

What's not clear (and maybe could be more of a concern) is the error may be a result of the possible problem I've mentioned previously about my system. I guess eventually one of us will be able to determine the source of this.

Thanks....cheers...

(2017-09-21 02:42)edit4ever Wrote:  I just ran a full day download with xtra details for all channels in the 94086 OTA lineup. Built the xmltv.xml file just fine.

I'm guessing the error is caused when details data is missing. I'll let this continue for a few days to see if the error shows up. Please let me know if you see an issue on your end.

Thanks!
find quote
mxlance Online
Donor
Posts: 405
Joined: May 2013
Reputation: 17
Post: #431
@edit4ever

have been somewhat following your progress with stephr1.
Question, is now the time to upgrade to latest version?
I am still using 0.7.1 no extra details. It is working well as is.
other than turning on extra details is not providing any extra details.

Also I have some time on and off for the next week if you need help trouble shooting.
find quote
Post Reply