Sindbad~EG File Manager
a
��gV+ � @ s d dl Z d dlZd dlZd dlZd dlZd dlmZ d dlm Z d dl
m Z m Z G dd� d�Z
G dd� de
ej�ZG d d
� d
e
ej�ZG d d � d e
ej�ZG d
d� de
ej�ZG dd� de
�ZG dd� deej�ZG dd� deej�ZG dd� de�ZG dd� de
ej�ZG dd� de
ej�ZG dd � d e
ej�ZG d d � d e
ej�ZG d d � d e�ZG d!d"� d"e
ej�Z G d#d$� d$e
ej�Z G d%d&� d&e
ej�Z G d'd(� d(e
ej�Z G d)d*� d*eej�Z G d+d,� d,e
ej�Z!G d-d.� d.e �Z"G d/d0� d0ej�Z#G d1d2� d2ej�Z$e%d3k�re�&� dS )4� N)�support)�
socket_helper)�BaseHTTPRequestHandler�
HTTPServerc @ sH e Zd ZdZdZg Zg ZdZdd� Zdd� Z dd � Z
d
d � Z d d
� Z dS )�
BaseRobotTest� Ztest_robotparserNc C s, t �| j��� }tj�� | _| j�|� d S �N) �io�StringIO�
robots_txt� readlines�urllib� robotparser�RobotFileParser�parser�parse)�self�lines� r �1/usr/local/lib/python3.9/test/test_robotparser.py�setUp s zBaseRobotTest.setUpc C s$ t |t�r|\}}||fS | j|fS r )�
isinstance�tuple�agent�r �urlr r r r �get_agent_and_url s
z BaseRobotTest.get_agent_and_urlc C s` | j D ]T}| �|�\}}| j||d��$ | �| j�||�� W d � q1 sP0 Y qd S �N)r r )�goodr �subTest�
assertTruer � can_fetchr r r r �test_good_urls s
z BaseRobotTest.test_good_urlsc C s` | j D ]T}| �|�\}}| j||d��$ | �| j�||�� W d � q1 sP0 Y qd S r )�badr r � assertFalser r! r r r r �
test_bad_urls# s
zBaseRobotTest.test_bad_urlsc C s | � | j�� | j� d S r )� assertEqualr � site_maps�r r r r �test_site_maps) s z BaseRobotTest.test_site_maps)
�__name__�
__module__� __qualname__r r r r# r' r r r" r% r) r r r r r s r c @ s e Zd ZdZddgZg d�ZdS )�UserAgentWildcardTestz�User-agent: *
Disallow: /cyberworld/map/ # This is an infinite virtual URL space
Disallow: /tmp/ # these will soon disappear
Disallow: /foo.html
�/�
/test.html)�/cyberworld/map/index.htmlz/tmp/xxx� /foo.htmlN�r* r+ r, r r r# r r r r r- - s r- c @ s e Zd ZdZg d�ZdgZdS )� CrawlDelayAndCustomAgentTestz�# robots.txt for http://www.example.com/
User-agent: *
Crawl-delay: 1
Request-rate: 3/15
Disallow: /cyberworld/map/ # This is an infinite virtual URL space
# Cybermapper knows where to go.
User-agent: cybermapper
Disallow:
)r. r/ )Z cybermapperr0 r0 Nr2 r r r r r3 8 s r3 c @ s& e Zd ZdZddgZdgZddgZdS )� SitemapTesta # robots.txt for http://www.example.com/
User-agent: *
Sitemap: http://www.gstatic.com/s2/sitemaps/profiles-sitemap.xml
Sitemap: http://www.google.com/hostednews/sitemap_index.xml
Request-rate: 3/15
Disallow: /cyberworld/map/ # This is an infinite virtual URL space
r. r/ r0 z7http://www.gstatic.com/s2/sitemaps/profiles-sitemap.xmlz2http://www.google.com/hostednews/sitemap_index.xmlN)r* r+ r, r r r# r' r r r r r4 I s
�r4 c @ s e Zd ZdZg Zg d�ZdS )�RejectAllRobotsTestz(# go away
User-agent: *
Disallow: /
)r0 r. �/tmp/Nr2 r r r r r5 Z s r5 c @ s e Zd ZdZdZdd� ZdS )�BaseRequestRateTestNc C s� | j }| j| j D ]�}| �|�\}}| j||d��z | �|�|�| j� |�|�}| �|| j� | jd ur�| �|t j
j � | �|j | jj � | �|j
| jj
� W d � q1 s�0 Y qd S r )r r r# r r r& � crawl_delay� request_rateZassertIsInstancer
r � RequestRateZrequestsZseconds)r r r r Zparsed_request_rater r r �test_request_rateh s(
���z%BaseRequestRateTest.test_request_rate)r* r+ r, r9 r8 r; r r r r r7 d s r7 c @ s e Zd ZdZdgZdS )�
EmptyFileTestr z/fooN)r* r+ r, r r r r r r r< � s r< c @ s4 e Zd ZdZdZej�dd�ZdZ dgZ
g d�Z dS ) � CrawlDelayAndRequestRateTestz�User-agent: figtree
Crawl-delay: 3
Request-rate: 9/30
Disallow: /tmp
Disallow: /a%3cd.html
Disallow: /a%2fb.html
Disallow: /%7ejoe/index.html
�figtree� � � )r> r1 )�/tmpz /tmp.html� /tmp/a.html� /a%3cd.html� /a%3Cd.htmlz /a%2fb.htmlz/~joe/index.htmlN) r* r+ r, r r r
r r: r9 r8 r r# r r r r r= � s r= c @ s e Zd ZdZdS )�DifferentAgentTestz FigTree Robot libwww-perl/5.04N�r* r+ r, r r r r r rF � s rF c @ s" e Zd ZdZdgZg d�ZdZdS )�InvalidRequestRateTestz�User-agent: *
Disallow: /tmp/
Disallow: /a%3Cd.html
Disallow: /a/b.html
Disallow: /%7ejoe/index.html
Crawl-delay: 3
Request-rate: 9/banana
rB )r6 rC rD rE z /a/b.htmlz/%7Ejoe/index.htmlrA N)r* r+ r, r r r# r8 r r r r rH � s rH c @ s e Zd ZdZdgZg ZdS )�InvalidCrawlDelayTestz2User-Agent: *
Disallow: /.
Crawl-delay: pears
r1 Nr2 r r r r rI � s rI c @ s e Zd ZdZdZdgZdgZdS )� AnotherInvalidRequestRateTestzeUser-agent: Googlebot
Allow: /folder1/myfile.html
Disallow: /folder1/
Request-rate: whale/banana
� Googlebot�/folder1/myfile.html�/folder1/anotherfile.htmlN�r* r+ r, r r r r# r r r r rJ � s rJ c @ s e Zd ZdZdZdgZdS )�UserAgentOrderingTestzMUser-agent: Googlebot
Disallow: /
User-agent: Googlebot-Mobile
Allow: /
rK z/something.jpgN)r* r+ r, r r r# r r r r rO � s rO c @ s e Zd ZdZdS )�UserAgentGoogleMobileTestzGooglebot-MobileNrG r r r r rP � s rP c @ s e Zd ZdZdZdgZdgZdS )�GoogleURLOrderingTestzJUser-agent: Googlebot
Allow: /folder1/myfile.html
Disallow: /folder1/
Z googlebotrL rM NrN r r r r rQ � s rQ c @ s e Zd ZdZdgZdgZdS )�DisallowQueryStringTestz2User-agent: *
Disallow: /some/path?name=value
�
/some/pathz/some/path?name=valueNr2 r r r r rR � s rR c @ s e Zd ZdZdgZdgZdS )� UseFirstUserAgentWildcardTestzNUser-agent: *
Disallow: /some/path
User-agent: *
Disallow: /another/path
z
/another/pathrS Nr2 r r r r rT � s rT c @ s e Zd ZdZdgZdgZdS )�EmptyQueryStringTestz>User-agent: *
Allow: /some/path?
Disallow: /another/path?
z /some/path?z/another/path?Nr2 r r r r rU � s rU c @ s0 e Zd ZdZej�dd�ZdZddgZ dgZ
dS ) �DefaultEntryTestzOUser-agent: *
Crawl-delay: 1
Request-rate: 3/15
Disallow: /cyberworld/map/
rA � � r. r/ r0 N) r* r+ r, r r
r r: r9 r8 r r# r r r r rV s
rV c @ s e Zd ZdZdZdd� ZdS )�StringFormattingTestz�User-agent: *
Crawl-delay: 1
Request-rate: 3/15
Disallow: /cyberworld/map/ # This is an infinite virtual URL space
# Cybermapper knows where to go.
User-agent: cybermapper
Disallow: /some/path
zxUser-agent: cybermapper
Disallow: /some/path
User-agent: *
Crawl-delay: 1
Request-rate: 3/15
Disallow: /cyberworld/map/c C s | � t| j�| j� d S r )r& �strr �expected_outputr( r r r �test_string_formatting) s z+StringFormattingTest.test_string_formattingN)r* r+ r, r r[ r\ r r r r rY s
rY c @ s e Zd Zdd� Zdd� ZdS )� RobotHandlerc C s | � dd� d S )Ni� zForbidden access)Z
send_errorr( r r r �do_GET/ s zRobotHandler.do_GETc G s d S r r )r �format�argsr r r � log_message2 s zRobotHandler.log_messageN)r* r+ r, r^ ra r r r r r] - s r] c @ s* e Zd Zdd� Zdd� Zejdd� �ZdS )� PasswordProtectedSiteTestCasec C sP | � tjj� ttjdft�| _t j
d| jj ddid�| _ d| j _
| j �� d S )Nr zHTTPServer servingZ
poll_intervalg{�G�z�?)�name�target�kwargsT)Z
addCleanupr
Zrequest�
urlcleanupr r �HOSTr] �server� threading�ThreadZ
serve_forever�t�daemon�startr( r r r r 8 s �z#PasswordProtectedSiteTestCase.setUpc C s" | j �� | j�� | j �� d S r )rh �shutdownrk �joinZ server_closer( r r r �tearDownH s
z&PasswordProtectedSiteTestCase.tearDownc C s\ | j j}dtj d t|d � }|d }tj�� }|�|� |� � | �
|� d|�� d S )Nzhttp://�:rX z /robots.txt�*) rh Zserver_addressr rg rZ r
r r Zset_url�readr$ r! )r �addrr Z
robots_urlr r r r �testPasswordProtectedSiteM s
z7PasswordProtectedSiteTestCase.testPasswordProtectedSiteN)r* r+ r, r rp r Z reap_threadsru r r r r rb 6 s rb c @ sF e Zd ZdZd�e�Zedd� �Zdd� Zdd� Z d d
� Z
d d � Z d
S )�NetworkTestCasezhttp://www.pythontest.net/z{}elsewhere/robots.txtc C sT t �d� t�| j��* tj�| j�| _ | j �
� W d � n1 sF0 Y d S )N�network) r Zrequiresr Ztransient_internet�base_urlr
r r r r rs )�clsr r r �
setUpClass] s
zNetworkTestCase.setUpClassc C s$ d� | j|tj�|�d s dnd�S )Nz{}{}{}rX r. r )r_ rx �os�path�splitext)r r| r r r r d s �zNetworkTestCase.urlc C sV | � | jj� | � | jj� | �| j�� d� | � | j�d�� | � | j�d�� d S )Nr rr )r$ r � disallow_all� allow_allZ
assertGreater�mtimer8 r9 r( r r r �
test_basici s
zNetworkTestCase.test_basicc C s� | � | j�d| �d��� | �| j�d| j�� | �| j�d| �d��� | �| j�d| �d��� | �| j�d| �d��� | � | j�d| j�� d S )Nrr Z elsewhereZNutchZbrianZwebstats)r r r! r r$ rx r( r r r �test_can_fetchp s z NetworkTestCase.test_can_fetchc C sf t j�| �d��}|�� | �|j� | �|j� | � |�
� d� | � |� d�� | � |�
d�� d S )Nz i-robot.txtr rr )r
r r r rs r r r$ r~ r& r� Z assertIsNoner8 r9 )r r r r r �
test_read_404x s z NetworkTestCase.test_read_404N) r* r+ r, rx r_ r � classmethodrz r r� r� r� r r r r rv X s
rv �__main__)'r r{ ri ZunittestZurllib.robotparserr
�testr Z test.supportr Z http.serverr r r ZTestCaser- r3 r4 r5 r7 r< r= rF rH rI rJ rO rP rQ rR rT rU rV rY r] rb rv r* �mainr r r r �<module> s@ "
")
Sindbad File Manager Version 1.0, Coded By Sindbad EG ~ The Terrorists