最近打算用爬虫爬取一些资料，主要是图片来扩充数据库 先用一些文字信息练手 还是第三方的requests好用呢，比python自带的urllib 好用呢
import requestsfrom bs4 import BeautifulSoupheaders = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36'} #伪装成浏览器访问for i in range(10):url='/top250?start='+str(i*25)response=requests.get(url,headers=headers)content=response.textsoup=BeautifulSoup(content,'lxml')items1=soup.find_all('div','pic')items2=soup.find_all('div','info')for item1,item2 in zip(items1,items2):index=item1.em.stringtitle=item1.a.img.get('alt')ifo=item2.find('div','bd').p.contents[2]ifo=ifo.replace(' ','')ifo=ifo.replace('\n','')print(index ,'\t' , title,'\t',ifo)
----------------------------------------------7.31 更新---------------------------------------------------------------------
今天有时间做了python 图片爬虫 爬取豆瓣top250电影的图片和电影名
import requestsfrom bs4 import BeautifulSoupimport cv2import osheaders = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36'}count=1if not os.path.exists('./pic'):os.mkdir('./pic')for i in range(10):url='/top250?start='+str(i*25)response=requests.get(url,headers=headers)content=response.textsoup=BeautifulSoup(content,'lxml')pics=soup.find_all('div','pic')for i,pic in enumerate(pics):img=pic.a.img.get('src')name=pic.a.img.get('alt')print('%d\t%s\t\t%s'%(count,name,img))cap=cv2.VideoCapture(img)if (cap.isOpened):ret,frame=cap.read() cv2.imencode('.jpg',frame)[1].tofile('./pic/%d.%s.jpg'%(count,name))count+=1#cv2.imwrite('./pic/%s.jpg'%name,frame) 由于想要添加的文件名含有中文，所以不能用imwrite 会有乱码 改用imencodecv2.waitKey(0)
运行代码后的效果如下 只需要不超过10s的时间，250张电影海报图片就轻松下载到本地PC上面啦
