Issues Installing Scrapy

[Positivity]

I am attempting to build my first bots with the help of scrapy, as I am most familiar with python and think it would be a good place to start, especially given the write-up in the DevOps section (@turbin3). However, I am having some issues installing the framework and have tried a few things to no avail. I have pip installed, and that didn't work. I even downloaded Miniconda to give that a go-- no luck.

Currently, this is what I am getting:

building 'lxml.etree' extension
error: Unable to find vcvarsall.bat

----------------------------------------
Failed building wheel for lxml

I'm out of ideas. Any help is much appreciated.

 

[CCarter]

Did you Google the error code mate? There looks like an easy Stackoverflow solution. What version of Python are you running?

 

[Positivity]

Did you Google the error code mate? There looks like an easy Stackoverflow solution. What version of Python are you running?

I found the source of the problem after some more digging. Apparently, I did not have the correct Wheel installed, which is required for pip to interact with certain installation packages.

That .whl file was contained within the lxml package that is apparently needed for Scrapy to function correctly, which makes sense because lxml is a library for processing html in python.

In order to figure out which package to use for my system, I ran python in the command prompt and ran the following code in order to see which tags pip allows (which will apparently be system dependent):

>>> import pip
>>> print(pip.pep425tags.get_supported())

Reviewing the tags, I was able to download the correct package, which then allowed me to install Scrapy. Cool.

 

[CCarter]

Cool. A younger harsher Carter would have roasted you for that one...

Back when I was coming up, we didn't have Google. Imagine 1995 trying to figure out error codes, had to walk 50 miles in the snow with no shoes to the library, this huge dusty building where it had books of information, and then figure out the Dewey Decimal Classification to fine the location of books, and then look in the back in the appendix of the book and if wasn't in there, god-forbid you had to read the whole book to figure out if the answer was even in there. Now its just stackoverflow.com - you young punks don't really understand how truly easy it is...

and get off my lawn!

 

[Positivity]

Now its just stackoverflow.com - you young punks don't really understand how truly easy it is...and get off my lawn!

It took some time to piece different things together, but I definitely couldn't imagine doing it without the available online resources.

 

[turbin3]

Sorry I missed this. Been crazy busy all week. How are the bots coming along? Any questions or hurdles to overcome? I'm trying to overcome one myself at the moment.

 

[Positivity]

Sorry I missed this. Been crazy busy all week. How are the bots coming along? Any questions or hurdles to overcome? I'm trying to overcome one myself at the moment.

Started building out an Instagram bot, but ran into problems with limits set by their API with regards to how many requests can be sent. Now I am working with twitter and trying to do some creative things as far as how the bot interacts with users.

What issue are you dealing with?

 

[SmokeTree]

Back when I was coming up, we didn't have Google. Imagine 1995 trying to figure out error codes

How about trying to figure out "how to program" with only 1 reference book available, a handful of magazines (with mostly bad code), NONE in the library, NO Internet and the only support was people on BBS systems, and most of those people weren't even coders? This was back in 1984 and I was even saving programs I wrote to cassette tape (yes you read that right). Still I learned BASIC and Commodore ML (Machine Lang) and here I am after learning over 30 langs and counting. Good 'ol trial and error and a ton of discipline

 

[Positivity]

How about trying to figure out "how to program" with only 1 reference book available, a handful of magazines (with mostly bad code), NONE in the library, NO Internet and the only support was people on BBS systems, and most of those people weren't even coders? This was back in 1984 and I was even saving programs I wrote to cassette tape (yes you read that right). Still I learned BASIC and Commodore ML (Machine Lang) and here I am after learning over 30 langs and counting. Good 'ol trial and error and a ton of discipline

That's pretty dope. I saw your recent post about Elixir. What would you tell a junior developer regarding picking a language? I have become proficient with Python and have begun to work with JS. Should I become an expert at these languages, or seek out more innovative languages such as Elixir?

 

[SmokeTree]

I have become proficient with Python and have begun to work with JS. Should I become an expert at these languages, or seek out more innovative languages such as Elixir?

I'd recommend sticking with what you are already familiar with. Both Python and JavaScript are excellent languages to start with and you can build just about anything you want by knowing just those 2. Learn them until you start thinking/dreaming in code/syntax. That's when the "Zen" part of coding starts to kick in. You'll wake up from dreams with completed code in your head. You'll start to view everyday tasks in a "how can I automate this?" way. A friend of mine, Zed Shaw wrote an excellent book on Python called "Learn Python the Hard Way". It's available for purchase but he also makes it available online: http://learnpythonthehardway.org/

Elixir isn't a language I would recommend to a new coder as a first lang because the syntax and concepts are different than just about any language but Erlang and Prolog. Once you're really comfortable with the Python and JavaScript way of doing things, Elixir would be on the top of my list of langs to learn next because of the performance, scalibility and fault tolerance. To compare Elixir to Python, I'd say that Python is a Ferrari and Elixir is an F14.